Information processing apparatus, information processing method, and program

Abstract

To improve usability of an operation of a virtual object in a virtual space.SOLUTION: An information processing apparatus comprises: an acquisition unit that acquires information indicating a rotation axis of a first virtual object in a virtual space that is moved on the basis of the position of a target in a real space, the rotation axis being selected on the basis of the position of the first virtual object; and a virtual object control unit that rotates, in the virtual space, the first virtual object around the rotation axis indicated by the information acquired by the acquisition unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, there have been many software that output images of virtual spaces and virtual objects (for example, three-dimensional objects) such as three-dimensional CAD (3 Dimension Computer Aided Design) to a display.

[0003] Generally, in software for operating such virtual spaces and virtual objects, a user can freely change the position and orientation of a virtual object in a three-dimensional space.

[0004] To change the position and orientation of a virtual object displayed on a display, various techniques have been developed. For example, Patent Document 1 discloses a technique for changing the orientation of a virtual object based on operations on two operation areas displayed on a display. More specifically, according to the technique disclosed in Patent Document 1, two-directional rotation is changed based on an operation on one of the three-directional rotations, and one-directional rotation is changed based on an operation on the other operation area.

[0005] Also, Non-Patent Document 1 discloses a technique for performing a total of six-directional movement operations including three-directional translation and three-directional rotation on a virtual object by operating a virtual hand displayed on a display to grab, move, and rotate the virtual object with the virtual hand.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Non-Patent Literature

[0007]

Non-Patent Literature 1

Summary of the Invention

Problems to be Solved by the Invention

[0008] However, in the technology disclosed in Patent Document 1, it is necessary to perform the operation while checking both the operation area and the virtual object, so there is a possibility that the user may feel annoyed by the operation. Further, in the technology disclosed in Non-Patent Literature 1, depending on the accuracy and measurement range of the controller or sensor used when operating the virtual hand, a situation may occur in which the virtual object rotates in an unexpected direction for the user.

[0009] Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a novel and improved technology capable of improving the usability of operations on virtual objects in a virtual space.

Means for Solving the Problems

[0010] In order to solve the above problems, according to an aspect of the present invention, there is provided an information processing apparatus including: an acquisition unit that acquires information indicating a rotation axis of a first virtual object in a virtual space, the rotation axis being selected based on the position of the first virtual object in the virtual space and being moved based on the position of an object in the real space; and a virtual object control unit that rotates the first virtual object around the rotation axis indicated by the information acquired by the acquisition unit within the virtual space.

[0011] The information processing apparatus further includes an option control unit that performs control to arrange options of the rotation axis on the same circumference centered on the position corresponding to the first virtual object in the virtual space within the virtual space, and the acquisition unit may acquire information indicating the rotation axis corresponding to the option when the first virtual object moves to the position of the option on the virtual space.

[0012] The option control unit may further perform control to arrange, in the virtual space, a display that guides the rotation direction around the rotation axis corresponding to the option that is closest to the first virtual object in the virtual space among the options of the rotation axis.

[0013] The virtual object control unit may execute a first control to move the position of the first virtual object based on the position of the object while fixing the posture of the first virtual object until the information indicating the rotation axis is acquired by the acquisition unit.

[0014] The first control may further include control to move the position of the first virtual object based on the position of the object while rotating the first virtual object around the rotation axis when the information indicating the rotation axis is acquired by the acquisition unit.

[0015] The virtual object control unit controls to arrange a second virtual object corresponding to the target in the virtual space, and the information processing apparatus further includes a determination unit that determines whether the states of the first virtual object and the second virtual object are in a first state or a second state. When the determination unit determines that the state is the first state, the virtual object control unit may execute the first control.

[0016] When the virtual object control unit determines that the states of the first virtual object and the second virtual object are in the second state by the determination unit, the virtual object control unit may control the position and orientation of the first virtual object according to the position and orientation of the target in the real space.

[0017] The determination unit may determine whether the state is the first state or the second state based on whether the first virtual object and the second virtual object are in contact with each other in the virtual space.

[0018] The target is the left and right hands of the user, and the second virtual object includes a left hand object corresponding to the user's left hand and a right hand object corresponding to the user's right hand. When the first virtual object is in contact with the left hand object and the right hand object, the determination unit may determine that the state is the first state.

[0019] When the first virtual object is in contact with only one of the left hand object and the right hand object, the determination unit may determine that the state is the second state.

[0020] When the determination unit determines that the first virtual object is not in contact with the left hand object and the right hand object, it determines that the state is the third state. When the virtual object control unit determines that the state is the third state by the determination unit, the virtual object control unit may execute control to fix the position and orientation of the first object.

[0021] The acquisition unit may further acquire information indicating the rotation direction around the rotation axis, and the virtual object control unit may rotate the first virtual object in the rotation direction indicated by the information acquired by the acquisition unit.

[0022] The acquisition unit may acquire information indicating the rotation direction based on the rotation direction on the real space of the target.

[0023] The acquisition unit may further acquire information indicating the rotation speed around the rotation axis, and the virtual object control unit may rotate the first virtual object at the rotation speed indicated by the information acquired by the acquisition unit.

[0024] The acquisition unit may acquire information indicating the rotation speed based on the rotation angle from the reference on the real space of the target.

[0025] Also, according to another aspect of the present invention to solve the above problems, obtaining information indicating the rotation axis of the first virtual object in the virtual space, which is selected based on the position of the first virtual object in the virtual space and is moved based on the position of the target in the real space, and rotating the first virtual object around the rotation axis indicated by the obtained information in the virtual space. An information processing method executed by a computer is provided.

[0026] Also, according to another aspect of the present invention for solving the above problems, there is provided a program that causes a computer to function as an acquisition unit that acquires information indicating a rotation axis of a first virtual object in a virtual space, the rotation axis being selected based on the position of the first virtual object in the virtual space and moved based on the position of an object in the real space, and a virtual object control unit that rotates the first virtual object around the rotation axis indicated by the information acquired by the acquisition unit in the virtual space.

Advantages of the Invention

[0027] As described above, according to the present invention, the usability of operations on virtual objects in a virtual space is improved.

Brief Description of the Drawings

[0028]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0029] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0030] <<1. Overview>> First, an overview of an embodiment of the present invention will be described. As described below, in this embodiment, a technique for improving the usability of operations on virtual objects in a virtual space is mainly proposed. The present invention is applicable, for example, to an information processing apparatus that provides an environment for practicing work by performing work on an object in an industrial site on a virtual object in a virtual space. However, it is applicable in various scenes where virtual objects are handled.

[0031] In this embodiment, based on the positions and postures of the user's both hands in the real space obtained from the sensor, the position and posture of the virtual object are determined. Here, based on the positions and postures of the user's both hands in the real space, the control method of the position and posture of the virtual object transitions. More specifically, in this embodiment, control of the virtual object with different degrees of freedom (DoF: Degree of Freedom) is realized for each control method. Thereby, the user can easily change the position and posture of the virtual object to a desired position and posture.

[0032] The number of dimensions (i.e., the number of movable directions) in which the position and posture of the virtual object can be simultaneously manipulated by the user in each control method is expressed as the degree of freedom in each control method. Hereinafter, such a degree of freedom is expressed by attaching "DoF" as a unit to the number representing the degree of freedom. For example, when the user can simultaneously manipulate the movement (translation) of the virtual object in the directions parallel to the X-axis, Y-axis, and Z-axis (X direction, Y direction, Z direction) in the XYZ coordinate system and the rotations (Roll, Pitch, Yaw) with the axes parallel to each of the X-axis, Y-axis, and Z-axis as the rotation axes, the number of dimensions that can be simultaneously manipulated is expressed as "6DoF".

[0033] One of the control methods for the virtual object according to this embodiment simultaneously realizes the operation of the virtual object with 6DoF including movement in the X direction, Y direction, and Z direction and rotations of Roll, Pitch, and Yaw.

[0034] Also, another one of the control methods for the virtual object according to this embodiment simultaneously realizes the operation of the virtual object with 4DoF including movement in the X direction, Y direction, and Z direction and any one rotation of Roll, Pitch, and Yaw.

[0035] In the operation of a virtual object in 4DoF, which rotation among Roll, Pitch, and Yaw is to be executed is determined based on the position of the virtual object in the virtual space. More specifically, by presenting a GUI (Graphical User Interface) for selecting the rotation axis of the virtual object to the user, the operability of selecting the rotation axis is improved.

[0036] <<2. Configuration Example>> Subsequently, a configuration example of the information processing apparatus according to the present embodiment will be described. FIG. 1 is a block diagram showing an example of the configuration of an information processing apparatus 10 according to the present embodiment. The information processing apparatus 10 is an information processing apparatus such as a PC (Personal Computer), a smartphone, an HMD (Head Mounted Display), etc., which is realized by a computer. As shown in FIG. 1, the information processing apparatus 10 includes a sensor unit 100, a control unit 200, a video output unit 300, and a storage unit 400.

[0037] (Sensor Unit 100) The sensor unit 100 has physical sensors, and detects sensor data of the user's hand by the physical sensors. The user's hand is an example of an object in the real space. Here, the type of the sensor is not particularly limited as long as it can acquire data that enables the hand tracking unit 210 described later to estimate the skeletons of both hands of the user and each finger of both hands. For example, the sensor may be an infrared sensor. In this case, the sensor data may be a Depth image. Alternatively, the sensor may be an RGB (Red, Green, Blue) camera or the like. In this case, the sensor data may be an RGB image.

[0038] Note that the sensor unit 100 may be configured as a device separate from the information processing apparatus 10, and in that case, the information processing apparatus 10 may acquire sensor data by communicating with the device.

[0039] (Control Unit 200) The control unit 200 performs various arithmetic operations. For example, the control unit 200 includes an arithmetic unit such as a CPU (Central Processing Unit), and its function can be realized by the program stored in the ROM (Read Only Memory) being expanded and executed in the RAM by the arithmetic unit. At this time, a computer-readable recording medium recording the program may also be provided. Alternatively, these blocks may be configured by dedicated hardware or may be configured by a combination of multiple hardware components.

[0040] Data required for the arithmetic operations by the arithmetic unit is appropriately stored by the storage unit 400 described later. Also, as shown in FIG. 1, the control unit 200 includes a hand tracking unit 210 and a virtual space control unit 220.

[0041] (Hand Tracking Unit 210) The hand tracking unit 210 estimates the skeletons of the user's hand and fingers based on the sensor data acquired from the sensor unit 100. More specifically, the hand tracking unit 210 may perform a process of measuring the position and posture of the user's hand and fingers in a coordinate system (hereinafter also referred to as the "sensor coordinate system") that defines the detection range of the real space by the sensor unit 100. Hereinafter, the user's hand and fingers are also referred to as "fingers".

[0042] The position of the fingers is represented by three-dimensional information in the sensor coordinate system. For example, the position of the fingers may be represented by three-dimensional coordinates in the sensor coordinate system of the positions of the joints of both hands.

[0043] The finger posture is represented by three-dimensional information in the sensor coordinate system. For example, the finger posture may be represented by a rotation matrix with respect to the coordinate axes in the sensor coordinate system in three directions along the finger posture. The three directions along the finger posture may be three mutually orthogonal directions. For example, the direction in which the palm faces, and two directions that are orthogonal to the direction in which the palm faces and orthogonal to each other may be set as the three directions along the finger posture. Note that the finger posture may be represented in a form such as Euler angles or quaternions.

[0044] Based on the result of skeleton estimation, the hand tracking unit 210 detects, as a predetermined gesture, that the shapes of the left hand and the right hand are in a predetermined shape. The predetermined gesture may be, for example, a state where the hand is open and a state where the hand is closed. The state where the hand is closed may be, for example, a state where all or any of the index finger, middle finger, ring finger, and little finger of the user's hand are in contact with the palm. The state where the hand is open may be a state other than the state where the hand is closed. For example, the state where the hand is open may be a state where any or all of the index finger, middle finger, ring finger, and little finger of the user's hand are not in contact with the palm. Hereinafter, the state where the hand is open and the state where the hand is closed are collectively referred to as "hand opening and closing".

[0045] Note that the predetermined gesture is not limited to hand opening and closing. For example, the predetermined gesture may be a gesture for the user to pinch an object or grasp an object. The gesture for pinching an object may be, for example, a state where the user's thumb and index finger are in contact.

[0046] (Virtual Space Control Unit 220) The virtual space control unit 220 generates a virtual space including a virtual object and a hand model. As shown in FIG. 1, the virtual space control unit 220 includes a hand model control unit 221, a virtual object control unit 222, a state determination unit 223, a UI control unit 224, and an image generation unit 225.

[0047] (Hand model control unit 221) The hand model control unit 221 functions as a virtual object control unit that generates a hand model in the virtual space using the detection results by the hand tracking unit 210. The hand model is a virtual hand in which the fingers of both hands of the user, whose skeleton is estimated by the hand tracking unit 210, are reproduced in the virtual space. The hand model is an example of a second virtual object. The hand model may be, for example, a virtual object imitating a hand. However, the hand model is not particularly limited as long as it is a virtual object that can recognize that the user is performing a predetermined gesture when the user performs a predetermined gesture by visually recognizing the hand model.

[0048] Hereinafter, the hand models that reproduce the left hand and the right hand of the user are also referred to as "left hand model" and "right hand model", respectively. The left hand model is an example of a left hand object. The right hand model is an example of a right hand object.

[0049] Also, hereinafter, when the user's hand is performing a predetermined gesture, it is assumed that the hand model also performs a predetermined gesture. Therefore, hereinafter, the opening and closing of the user's hand is also referred to as the opening and closing of the hand model.

[0050] The hand model control unit 221 may input the posture of the user's fingers estimated by the hand tracking unit 210 to each joint of the virtual fingers constituting the hand model. Thereby, the hand model control unit 221 can move the hand model in the same way as the user's fingers. Also, the hand model control unit 221 can create a hand model by arranging a figure such as a sphere on the virtual space according to the estimated finger positions and connecting the figures with a cylinder according to the result of the skeleton estimation.

[0051] The hand model control unit 221 controls to arrange the hand model in the virtual space by outputting information about the generated hand model to the image generation unit 225.

[0052] (Virtual Object Control Unit 222) The virtual object control unit 222 controls virtual objects in the virtual space. A virtual object is an object that is operated by a user in the virtual space and is an example of a first virtual object. The shape and the like of the virtual object are not particularly limited. For example, as the virtual object, an object to be worked on at the work site may be reproduced.

[0053] The virtual object control unit 222 controls the position and orientation of the virtual object based on the position and orientation of the hand model generated by the hand model control unit 221. Note that the virtual object control unit 222 controls the position and orientation of the virtual object by a control method determined based on the states of the virtual object and the hand model obtained from the state determination unit 223 described later. Here, the control method of the virtual object by the virtual object control unit 222 includes control in 6DoF and control in 4DoF. Details of the control method will be described later.

[0054] The virtual object control unit 222 controls to arrange the virtual object in the virtual space by outputting information about the generated virtual object to the image generation unit 225.

[0055] (State Determination Unit 223) The state determination unit 223 functions as a determination unit that determines the states of the virtual object and the hand model based on the control content of the hand model by the hand model control unit 221 and the control content of the virtual object by the virtual object control unit 222.

[0056] The states of the virtual object and the hand model may include, for example, the relationship between the virtual object and the hand model. The relationship between the virtual object and the hand model may be whether or not the virtual object and the hand model are in contact on the virtual space.

[0057] In addition, the states of the virtual object and the hand model may include the respective states of the virtual object and the hand model. More specifically, the state of the hand model may indicate whether or not a predetermined gesture detected by the hand model control unit 221 is being performed.

[0058] The state determination unit 223 notifies the virtual object control unit 222 of the determination result of the states of the virtual object and the hand model.

[0059] (UI control unit 224) The UI control unit 224 controls the output of a UI (User Interface) for selecting the rotation axis of the virtual object in the operation of the virtual object in 4DoF. Hereinafter, such a UI is also referred to as a "rotation axis UI". More specifically, the UI control unit 224 functions as a selection option control unit that controls the output of a rotation axis UI including rotation axis options for acquiring information indicating the rotation axis of the virtual object. More specifically, the rotation axis options include options for Roll, Pitch, and Yaw. Hereinafter, an example in the case where such a UI is a GUI displayed on the video output unit 300 described later will be mainly described.

[0060] The UI control unit 224 may execute control to display the rotation axis UI on the video output unit 300 described later when the control method transitions to 4DoF control. Further, when the rotation axis of the virtual object is selected based on the operation of the rotation axis UI, the UI control unit 224 may execute control to hide the rotation axis UI from the video output unit 300.

[0061] In addition, the UI control unit 224 functions as an acquisition unit that acquires rotation axis information, which is information indicating the rotation axis of the virtual object selected based on the operation of the position of the virtual object by the user.

[0062] (Image generation unit 225) The image generation unit 225 generates an image of a virtual space including virtual objects and hand models. Here, it is mainly assumed that the image is mainly a moving image (video). The video generated by the image generation unit 225 is output to the video output unit 300 described later and thus made visible to the user. The user operates the virtual object by moving their own hand while checking the video to operate the hand model.

[0063] The image generation unit 225 generates a video of the virtual object and the hand model as seen from a certain viewpoint within the virtual space. The viewpoint may be the first-person viewpoint of the user. Thereby, the operability of the virtual object is improved. Note that the viewpoint may be arbitrarily changeable by the user.

[0064] (Video output unit 300) The video output unit 300 is constituted by a physical display and has a function of displaying the video generated by the image generation unit 225.

[0065] For example, the video output unit 300 may be a liquid crystal display (LCD) device or an OLED (Organic Light Emitting Diode) device.

[0066] For example, the video output unit 300 may be constituted by a two-dimensional display. At this time, the video output unit 300 may display a two-dimensional video in which the three-dimensional virtual space generated by the image generation unit 225 is represented two-dimensionally. Alternatively, the video output unit 300 may be constituted by a three-dimensional display enabling stereoscopic viewing by the user. At this time, the video output unit 300 can display a three-dimensional video of the three-dimensional virtual space generated by the image generation unit 225. Since the video output unit 300 is constituted by a three-dimensional display, it is expected that the user can easily recognize the depth information of the virtual object, making it easier to operate the virtual object.

[0067] (Storage unit 400) The storage unit 400 is a memory capable of storing programs and data for operating the video output unit 300. Further, the storage unit 400 can also temporarily store various data required in the process of operating the video output unit 300. For example, the storage unit 400 may be a non-volatile memory.

[0068] <<3. Technical Problem>> In the technology described in Patent Document 1 above, it is cited as a problem that the number of dimensions in which virtual objects can be simultaneously manipulated is only 1 DoF. As a result, compared with the case where the number of dimensions in which simultaneous manipulation is possible is 2 DoF or more, the operation time of virtual objects by the user becomes longer.

[0069] On the other hand, in the technology described in Non-Patent Document 1 above, since the number of dimensions in which virtual objects can be simultaneously manipulated is 6 DoF, the operation on virtual objects becomes closer to the handling of objects in the real space. However, depending on the accuracy and measurement range of the controller or sensor used when operating the virtual hand, a situation may occur where the virtual object rotates in an unexpected direction for the user. Therefore, when performing practice of complex work involving fine posture changes of an object at a work site by operating a virtual object that reproduces the object, the technology described in Non-Patent Document 1 is not suitable.

[0070] Non-Patent Document 1 also discloses a technology for displaying a GUI for operating virtual objects. The GUI disclosed in Non-Patent Document 1 is a GUI generally called a "gizmo". Gizmos are commonly used in software such as CAD that handles three-dimensional objects. A gizmo includes displays of the X-axis, Y-axis, and Z-axis set for the virtual space. The user can select an axis indicating the moving direction of the virtual object or the rotation axis of the virtual object by operating the hand model so as to grasp each axis. Then, the user operates the position and orientation of the virtual object by moving or rotating the hand model that has grasped the axis.

[0071] FIG. 2 is a diagram for explaining an example of an operation of a virtual object using a gizmo. FIGS. 2 show display screens D1, D2, and D3 that are screens displayed to a user in the operation of the virtual object. The display screen D1 represents a virtual space including a virtual object O1 and a hand model H1. The display screen D1 also includes an axis AX1, an X-direction operation axis Ax, a Y-direction operation axis Ay, and a Z-direction operation axis Az. The axis AX1 is a display for guiding the X-axis, Y-axis, and Z-axis in the XYZ coordinate system set for the virtual space. The X-direction operation axis Ax, the Y-direction operation axis Ay, and the Z-direction operation axis Az are axes for operating the virtual object O1 that are parallel to the X-axis, Y-axis, and Z-axis, respectively, in the XYZ coordinate system set for the virtual space.

[0072] For example, as shown in the display screen D2, the user can move the hand model H1 in the moving direction Dm1 while grasping the X-direction operation axis Ax with the hand model H1, thereby changing the position of the virtual object O1 on the display screen D1 to a position moved in the moving direction Dm1 parallel to the X-axis.

[0073] Further, as shown in the display screen D3, the user can rotate the hand model H1 in the rotation direction Dr1 with the Y-axis as the rotation axis while grasping the Y-direction operation axis Ay with the hand model H1, thereby changing the posture of the virtual object O1 on the display screen D1 to a posture rotated in the rotation direction Dr1.

[0074] As described above, according to the operation of the virtual object using the gizmo, axes for operating one DoF at a time can be selected, and a total of 6 DoF operations can be sequentially performed on the virtual object. According to the operation of the virtual object using the gizmo, it is shown in Non-Patent Document 1 that the operation accuracy is improved by selecting an axis and operating one DoF at a time. However, according to such a technique, since the virtual object can be operated only one DoF at a time, the operation time of the virtual object by the user becomes longer compared to the case where the number of dimensions that can be operated simultaneously is 2 DoF or more.

[0075] For the purpose of solving these problems, the information processing apparatus 10 according to this embodiment has been developed. Hereinafter, the characteristic configuration of the information processing apparatus 10 for solving such problems will be described.

[0076] <<4. Characteristic Configuration of the Present Invention>> (1) Transition of the Control Method of Virtual Objects First, the transition of the control method of the virtual object by the virtual object control unit 222 based on the determination result of the state of the virtual object and the hand model by the state determination unit 223 will be described.

[0077] The state determination unit 223 according to this embodiment transitions the states of the virtual object and the hand model among five states. FIG. 3 is a diagram for explaining an example of the state transition of each state of the virtual object and the hand model. As shown in FIG. 3, the states of the virtual object and the hand model transition between states A to E respectively.

[0078] Each state transitions triggered by the contact state between each of the left - hand hand model and the right - hand hand model and the virtual object, and the opening and closing of each of the left - hand hand model and the right - hand hand model. For example, when the closed hand model contacts the virtual object, the state transitions from state A to any of states B to E. According to such a configuration, the user can intuitively pick up and operate the virtual object.

[0079] Hereinafter, the state where the hand model is in contact with the virtual object is also simply referred to as the "contact state". Also, the state where the hand model is not in contact with the virtual object is also simply referred to as the "non-contact state". Further, the state where the hand model is closed is also referred to as the "closed state". Further, the state where the hand model is open is also referred to as the "open state". Note that when the left hand model or the right hand model does not exist in the video generated by the image generation unit 225 (for example, when only one hand of the user is detected by the sensor unit 100, etc.), the hand model that does not exist in the video may be determined to be in the open state.

[0080] State A is the initial state of the states of the virtual object and the hand model, and is the state of "No Status" which does not correspond to other states. Also, state A is an example of the third state that is transitioned from other states when the right hand model and the left hand model are in the open state and the non-contact state.

[0081] State B is the state of "One Hand (L)" that is transitioned from other states when only the left hand model is in the closed state and the contact state. In state B, the left hand model is mainly involved in control. State C is the state of "One Hand (R)" that is transitioned when only the right hand model is in the closed state and the contact state. In state C, the right hand model is mainly involved in control. Note that state B and state C are examples of the first state.

[0082] States D and E are the states of "Both Hands (L)" and "Both Hands (R)" that are transitioned from other states when both hand models are in the closed state and the contact state. In state D, the left hand model is mainly involved in control. In state E, the right hand model is mainly involved in control. Which state the state transitions to, either state D or state E, is determined according to the hand model that first becomes in the closed state and the contact state. States D and E are examples of the second state.

[0083] When there is a hand model that has newly entered the closed and contact state, or when there is a hand model that has newly entered the open and non-contact state, a transition occurs from states B to E to other states. Note that the conditions for transitioning from state B to state D or from state C to state E do not necessarily require both hand models to be in the closed and contact state. More specifically, the conditions for these cases may be that at least a hand model different from the hand model mainly involved in control (for example, the right hand model in state B) transitions to the closed and contact state.

[0084] The specific transition conditions between each state will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the specific transition conditions between each state. In FIG. 4, the transition conditions between each of states A to E are shown. However, when transitions such as from state B to state E are not assumed, the description is omitted. For example, for transition condition number 1, the transition condition for transitioning from state A to state B is shown as the left hand being in the closed state and the left hand model being in the contact state.

[0085] Based on the control status of the hand model by the hand model control unit 221 and the control status of the virtual object by the virtual object control unit 222, the state determination unit 223 determines whether the transition conditions shown in FIG. 4 are satisfied. More specifically, the state determination unit 223 determines whether the transition conditions are satisfied based on the information on the position and orientation of the hand model obtained from the hand model control unit 221 and the information on the position and orientation of the virtual object obtained from the virtual object control unit 222. The state determination unit 223 outputs the determination result to the virtual object control unit 222 and the UI control unit 224.

[0086] The virtual object control unit 222 switches the control state of the virtual object based on the determination result by the state determination unit 223. For example, when the states of the virtual object and the hand model are state D or state E, the virtual object control unit 222 executes 4DoF control for the virtual object. However, until the rotation axis of the virtual object is selected based on the operation on the rotation axis UI, the virtual object control unit 222 executes 3DoF control for the virtual object. Here, the 3DoF control of the virtual object is control that moves only the position while fixing the posture of the virtual object. That is, in the 3DoF control of the virtual object, only the three-direction parallel movement of the virtual object is operated by the user.

[0087] When the states of the virtual object and the hand model transition to state D or state E, the UI control unit 224 controls the video output unit 300 so that the rotation axis UI is displayed. Then, when the selection of the rotation axis is accepted by the UI control unit 224, the virtual object control unit 222 executes 4DoF control for the virtual object. More specifically, when the selection of the rotation axis is accepted, the virtual object control unit 222 executes rotation around the selected rotation axis in addition to the 3DoF control. The control for the virtual object that is executed by switching between 3DoF control and 4DoF control in this way is also referred to as "first control". The details of the first control will be described later.

[0088] Also, when the states of the virtual object and the hand model transition to state B or state C, the virtual object control unit 222 executes 6DoF control for the virtual object. The 6DoF control is also referred to as "second control". The details of the second control will be described later.

[0089] In addition, when the states of the virtual object and the hand model are in state A, the virtual object control unit 222 executes control to fix the position and orientation of the virtual object, that is, control that does not change the position and orientation. Note that the control for fixing the position and orientation of the virtual object is also referred to as "third control".

[0090] (2) Each control method of the virtual object (First control) Subsequently, each control method (first control and second control) of the virtual object will be described. First, the first control will be described with reference to FIGS. 5 and 6. FIGS. 5 and 6 are diagrams for explaining an example of the operation of the virtual object by the first control.

[0091] When the control of the virtual object by the virtual object control unit 222 transitions from the second control or the third control to the first control (that is, when the states of the virtual object and the hand model transition from state A, state B, or state C to state D or state E), the UI control unit 224 controls to display the rotation axis UI on the video output unit 300.

[0092] FIG. 5 shows an example of the display screen D4 displayed on the video output unit 300 when the control of the virtual object by the virtual object control unit 222 transitions from the second control or the third control to the first control. The display screen D4 includes the virtual object O2, the left hand model HL1 and the right hand model HR1, the rotation axis options C1 to C3, and the axis AX2. As shown in the display screen D4, when the left hand model HL1 and the right hand model HR1 transition to the closed state and are in contact with the virtual object O2, the first control is started.

[0093] Axis AX2 is a display for guiding the X-axis, Y-axis, and Z-axis in the XYZ coordinate system set for the virtual space. As shown in FIG. 5, the X-axis, Y-axis, and Z-axis indicated by axis AX2 may be set to be in the left-right direction, up-down direction, and front-back direction with respect to the user when the user views the display screen D4 in the direction in which it is assumed that the user views the display screen D4. However, each axis is not limited to this example and may be set arbitrarily.

[0094] The rotation axis options C1 to C3 are an example of the rotation axis UI. The rotation axis options C1 to C3 are options for selecting Roll, Pitch, and Yaw, respectively, with axes parallel to the Y-axis, X-axis, and Z-axis indicated by axis AX2 as the rotation axes. In the following, when the rotation axis options C1 to C3 are not particularly distinguished, they are also simply referred to as "rotation axis option C".

[0095] The UI control unit 224 may control the rotation axis options C1 to C3 to be arranged on the same circumference centered on the position corresponding to the virtual object O2. The position corresponding to the virtual object O2 may be, for example, the position of the center of gravity of the virtual object O2. For example, in the display screen D4, an example is shown in which the rotation axis options C1 to C3 are arranged on a concentric circumference with a radius d1 centered on the position of the center of gravity of the virtual object O2. The radius d1 of the circle may be a predetermined value, or may be dynamically set so that the rotation axis options C1 to C3 do not overlap the virtual object O2.

[0096] Also, FIG. 5 shows an example in which the rotation axis options C1 to C3 are arranged at respective positions where the half side in the positive direction of the Z-axis set in the virtual space on the concentric circumference with a radius d1 is divided into three, but the arrangement position is not particularly limited. For example, the rotation axis options C1 to C3 may be arranged on the half side in the negative direction of the Z-axis, or the circumference may be divided into three and the rotation axis options C1 to C3 may be arranged.

[0097] Also, the rotation axis options C1 to C3 may be arranged continuously on a concentric circumference as shown in FIG. 5, or may be arranged separately. Further, the rotation axis options C1 to C3 are not limited to being arranged on a concentric circumference, and may be arranged at any position on the video output unit 300. Also, the rotation axis options C1 to C3 may not be displayed on the video output unit 300, and the user may be informed by voice as to which rotation axis can be selected by moving in which direction and to what extent.

[0098] When the virtual object O2 moves to any position among the rotation axis options C1 to C3 in the virtual space, the UI control unit 224 acquires rotation axis information, which is information indicating the rotation axis corresponding to the rotation axis option C1 to C3 at that position. The movement of the virtual object O2 to any position among the rotation axis options C1 to C3 in the virtual space may be, for example, that a part of the virtual object O2 comes into contact with a part of the rotation axis options C1 to C3. Hereinafter, an example in which the movement of the virtual object O2 to any position among the rotation axis options C1 to C3 in the virtual space is mainly described in the case where a part of the virtual object O2 comes into contact with a part of the rotation axis options C1 to C3. Alternatively, the movement of the virtual object O2 to any position among the rotation axis options C1 to C3 in the virtual space may be that the center of gravity position of the virtual object O2 comes into contact with a part of the rotation axis options C1 to C3.

[0099] The UI control unit 224 outputs the acquired rotation axis information to the virtual object control unit 222. Here, in the first control, the virtual object control unit 222 executes the control of the virtual object O2 in 3DoF until the rotation axis information is acquired by the virtual object control unit 222. More specifically, in the control of the virtual object O2 in 3DoF, the virtual object control unit 222 moves the position of the virtual object based on the position of the hand model (that is, based on the position of the user's hand in the real space) while fixing the posture of the virtual object.

[0100] More specifically, the virtual object control unit 222 moves the position of the virtual object O2 in the virtual space so as to follow one of the left hand model HL1 and the right hand model HR1. More specifically, when the states of the virtual object O2 and the hand model are in state D, the virtual object control unit 222 moves the position of the virtual object O2 so as to follow the left hand model HL1. Further, when the states of the virtual object O2 and the hand model are in state E, the virtual object control unit 222 moves the position of the virtual object O2 so as to follow the right hand model HR1. By controlling in this way, since the virtual object O2 follows the hand model that first touches the virtual object O2, the user can move the position of the virtual object O2 more intuitively.

[0101] The user moves the virtual object O2 in the virtual space by moving the hand model that the virtual object O2 follows by moving his or her hand. Then, the user can select the rotation axis of the virtual object O2 by moving the virtual object O2 so that the virtual object O2 contacts any one of the rotation axis options C1 to C3. That is, the user can rotate the virtual object O2 in the direction in which the user wants to rotate the virtual object O2 by bringing the virtual object O2 into contact with any one of the rotation axis options C1 to C3.

[0102] The display screen D5 shown in FIG. 5 is an example of a display screen when the user selects Roll as the rotation axis of the virtual object and transitions from the display screen D4. As shown in the display screen D5, within the virtual space shown on the display screen D5, the user moves the hand model (left hand model HL1 or right hand model HR1) followed by the virtual object O2 in the moving direction Dm2 by moving their own hand, thereby bringing the hand model into contact with the rotation axis option C2. As a result, the UI control unit 224 accepts, as the rotation axis, an axis parallel to the X-axis corresponding to the rotation axis option C2, that is, Roll. For example, the UI control unit 224 may accept, as the rotation axis, an axis parallel to the X-axis passing through the position corresponding to the virtual object O2 (here, the center of gravity position of the virtual object O2).

[0103] Further, the UI control unit 224 may execute control to cause the video output unit 300 to display a guidance display, which is a display for guiding the rotation direction of the virtual object O2 as the rotation axis UI. For example, the UI control unit 224 may display a guidance display for guiding the rotation direction around the rotation axis corresponding to the rotation axis option C that is closest to the virtual object O2 in the virtual space among the rotation axis options C1 to C3.

[0104] FIG. 6 shows an example of the display screen D6 when the display for guiding the rotation direction around the rotation axis is included in the rotation axis UI. The display screen D6 includes the virtual object O2, the left hand model HL1 and the right hand model HR1, the rotation axis options C1 to C3, and the guidance display Gr. As shown in FIG. 6, the UI control unit 224 displays a guidance display Gr for guiding the Roll direction, which is the rotation direction around the rotation axis corresponding to the rotation axis option C2 that is closest to the virtual object O2 in the virtual space among the rotation axis options C1 to C3. By displaying the guidance display Gr, when the user operates the left hand model HL1 and the right hand model HR1 to bring the virtual object O2 closer to the rotation axis option C2, the rotation direction of the virtual object O2 in the 4DoF control can be notified in advance.

[0105] When the UI control unit 224 accepts the selection of the rotation axis, the virtual object control unit 222 acquires the rotation axis information from the UI control unit 224. Then, the virtual object control unit 222 switches the control of the virtual object O2 in 3DoF to the control of the virtual object O2 in 4DoF. That is, the virtual object control unit 222 moves the position of the virtual object O2 based on the position of the hand model while rotating the virtual object O2 around the rotation axis represented by the rotation axis information. The virtual object control unit 222 may rotate the virtual object O2 in a predetermined direction (positive or negative direction of Roll, Pitch, or Yaw) and at a predetermined rotation speed. The predetermined direction and the predetermined rotation speed may be preset by the user.

[0106] Also, when the UI control unit 224 accepts the selection of the rotation axis, the UI control unit 224 executes control to make the rotation axis options C1 to C3 displayed on the video output unit 300 non-displayed. When the UI control unit 224 accepts the selection of the rotation axis, the guidance display Gr may also be made non-displayed, or the guidance display Gr may be continuously displayed. By continuously displaying the guidance display Gr, it becomes easier for the user to understand in which direction the virtual object O2 is rotating.

[0107] After the UI control unit 224 accepts the selection of the rotation axis, the virtual object control unit 222 continues to rotate the virtual object O2 until the states of the virtual object and the hand model transition from state D or state E to other states. When the states of the virtual object and the hand model transition from state D or state E to other states, since the control method switches from the first control to the second control or the third control, the rotation of the virtual object O2 around the rotation axis received by the UI control unit 224 stops.

[0108] After the rotation of the virtual object O2 around the rotation axis starts, the user can control the rotation time of the virtual object O2 by operating the hand model to transition the states of the virtual object and the hand model from state D or state E to other states. Thereby, the user can control the amount of rotation of the virtual object O2 around the rotation axis. By controlling the amount of rotation of the virtual object O2 in this way, the user can accurately change the posture of the virtual object O2 to a desired posture.

[0109] (Second Control) Subsequently, the second control of the virtual object will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining an example of the operation of the virtual object by the second control. FIG. 7 shows display screens D7 and D8 in the operation of the virtual object in 6DoF.

[0110] The display screens D7 and D8 include the virtual object O3 and the right hand model HR2. When only the right hand model HR2 transitions to the closed state and contacts the virtual object O3 as in the display screen D7, the second control is started because the states of the virtual object O3 and the hand model transition to state C.

[0111] In the second control, the virtual object control unit 222 controls the position and posture of the virtual object O3 according to the position and posture of the hand in the real space. More specifically, the virtual object control unit 222 moves the position and posture of the virtual object O3 in the virtual space so that the virtual object O3 follows the hand model.

[0112] For example, when the state of the virtual object O3 and the hand model is state C, the virtual object control unit 222 moves the position of the virtual object O3 so as to follow the right hand model HR2. The user can change the position and orientation of the virtual object O3 on the display screen D7 to a position moved in the movement direction Dm3 parallel to the X axis and an orientation rotated in the rotation direction Dr2 with the Y axis as the rotation axis by, for example, moving the right hand model HR2 in the movement direction Dm3 and rotating it in the rotation direction Dr2 as shown on the display screen D8.

[0113] Here, the second control when the state of the virtual object O3 and the hand model is state C has been described. However, when the state of the virtual object O3 and the hand model is state B, the position and orientation of the virtual object O3 are controlled to follow the left hand model.

[0114] The above describes each control method of the virtual object. By switching each control method described so far, the user can perform the operation desired for the virtual object more easily and accurately. For example, when the user wants to change the orientation of the virtual object more accurately, the user can operate the hand model so that the first control can be executed to change the orientation of the virtual object. Also, when the user wants to intuitively change the position and orientation of the virtual object, the user can operate the hand model so that the second control can be executed to change the position and orientation of the virtual object.

[0115] <<5. Operation Example>> Subsequently, an operation example of the information processing apparatus 10 according to the present embodiment will be described. FIG. 8 is a flowchart showing an operation example of the information processing apparatus 10 in an embodiment of the present invention.

[0116] First, the sensor unit 100 acquires sensor data of both hands of the user (S101). Subsequently, the hand tracking unit 210 analyzes the sensor data obtained by the sensor unit 100 to estimate the positions and postures of the joints of the left and right hands (S102).

[0117] The hand model control unit 221 operates the hand model in the virtual space according to the estimation result by the hand tracking unit 210 (S103). Subsequently, the state determination unit 223 determines the states of the virtual object and the hand model based on the control content of the hand model by the hand model control unit 221 and the control content of the virtual object by the virtual object control unit 222 (S104).

[0118] Then, the virtual object control unit 222 executes any one of the first control, the second control, and the third control according to the determination result of the states of the virtual object and the hand model by the hand model control unit 221, and controls the position and posture of the virtual object (S105).

[0119] Subsequently, the virtual space control unit 220 determines whether an end operation has been performed by the user (S106). The end operation may be, for example, the hand model contacting an end button displayed on the video output unit 300. If the end operation is performed by the user (S106 / YES), the process ends. If the end operation is not performed by the user (S106 / NO), the operations of S101 to S105 are repeated until the end operation is performed by the user.

[0120] Subsequently, the details of the process of S105 shown in FIG. 8 will be described with reference to FIG. 9. FIG. 9 is a flowchart showing a subroutine of the process of S105 shown in FIG. 8. When the process of S105 starts, first, the virtual object control unit 222 determines whether to execute the first control (S201).

[0121] When the virtual object control unit 222 executes the first control according to the determination result of the state of the virtual object and the hand model determined in S104 (S201 / YES), the virtual object control unit 222 makes the position of the virtual object follow the hand model and perform a parallel movement, and executes a simultaneous operation in 3DoF (S202). Then, the UI control unit 224 determines whether the rotation axis of the virtual object has been selected (S203).

[0122] When the rotation axis of the virtual object has not been selected (S203 / NO), the UI control unit 224 controls the video output unit 300 to display a rotation axis UI including rotation axis options C1 to C3 (S204). Subsequently, the UI control unit 224 determines whether the rotation axis has been selected (S205). More specifically, the UI control unit 224 detects that the virtual object has come into contact with any of the rotation axis options C1 to C3 as the selection of the rotation axis.

[0123] When the rotation axis has been selected (S205 / YES), the UI control unit 224 makes the rotation axis UI displayed on the video output unit 300 invisible (S206), and the process proceeds to S207. Also, when the rotation axis of the virtual object has already been selected (S203 / YES), the process proceeds to S207. On the other hand, when the rotation axis is not selected (S205 / NO), the process proceeds to S106 in the flowchart of FIG. 8.

[0124] The virtual object control unit 222 rotates the virtual object around the selected rotation axis according to the rotation axis information obtained from the UI control unit 224 (S207). Then, the process proceeds to S106. Thereby, the control of the virtual object in 4DoF is realized.

[0125] On the other hand, when the virtual object control unit 222 does not execute the first control (S201 / NO), the process proceeds to S208. Then, when the virtual object control unit 222 executes the second control according to the determination results of the states of the virtual object and the hand model determined in S104 (S208 / YES), the virtual object control unit 222 causes the virtual object to follow the hand model and performs parallel movement and rotational movement, and executes simultaneous operation in 6DoF (S209). Then, the process proceeds to S106.

[0126] When the virtual object control unit 222 does not execute the second control (S208 / NO), the virtual object control unit 222 executes the third control to fix the position and orientation of the virtual object in the virtual space (S210).

[0127] By executing the processes described above with reference to FIGS. 8 and 9, the user can operate the virtual object while switching between the first control, the second control, and the third control.

[0128] <<6. Example of operational effects>> According to the information processing apparatus 10 described above, based on the contact state between the virtual object and the hand model and the open / closed state of the hand model, the control method of the virtual object is transitioned, and in the first control, by allowing the user to select the rotation axis, the degrees of freedom of operation for each rotation are separated, thereby improving the accuracy of the attitude operation of the virtual object.

[0129] Also, in the second control, since simultaneous operation in 6DoF is possible, it is superior in terms of operation speed to the operation of the virtual object using a gizmo shown in Non-Patent Document 1.

[0130] Also, when comparing the control method of the virtual object according to the present embodiment and the control method of the virtual object by a gizmo in accordance with Fitts' law, it is led that the control method of the virtual object according to the present embodiment is superior in terms of ease of operation.

[0131] Fitts' law is a model that indicates the time taken to operate a pointer (in this embodiment, a hand model) and select a specific object. In Fitts' law, the index of difficulty "ID" to reach the target is expressed by the following formula (1). Here, reaching the target means selecting the rotation axis of the virtual object.

[0132]

Number

[0133] In formula (1), W represents the width of the option. Also, in formula (1), D is the distance from the selection start point to the object to be selected. Referring to formula (1), it can be seen that the larger W is and the smaller D is, the smaller ID becomes. That is, formula (1) indicates that the larger the size of the selection target and the shorter the distance to the selection target, the lower the difficulty of selection.

[0134] Here, by applying formula (1) to the rotation axis options C1 to C3 described using the gizmo and FIG. 5, the magnitudes of the ID, which are the indices of difficulty to reach each target, are compared. FIG. 10 is a diagram for explaining the magnitude of the index of difficulty "ID" to reach the target when controlling a virtual object with the gizmo.

[0135] FIG. 10 shows the X-axis operation axis Ax, Y-axis operation axis Ay, and Z-axis operation axis Az in the gizmo, and the right hand model HR3 for operating each operation axis. As shown in FIG. 10, when applying Equation (1) to the gizmo, the value of W varies depending on the positional relationship between the virtual object O4 and the right hand model HR3. However, when operating the virtual object O4 in the gizmo, it is considered that the radius R, which is the diameter of each of the X-axis operation axis Ax, Y-axis operation axis Ay, and Z-axis operation axis Az, reaches the maximum value. To decrease the value of ID, it is desirable to increase the diameter R of each axis. However, when the diameter R of each axis is increased, each axis may hide the virtual object O4, so there is a limit to the size. Also, D in Equation (1) has different values depending on the axis selected. For example, when selecting the Y-axis operation axis Ay, D in Equation (1) is d4. That is, when applying Equation (1) to the gizmo, the calculated ID has different values depending on the axis selected by the user.

[0136] On the other hand, as shown in FIG. 5, the rotation axis options C1 to C3 are arranged on the same circumference with a radius d1 centered at the position corresponding to the virtual object O2. Therefore, when applying Equation (1) to the rotation axis options C1 to C3, D in Equation (1) is d1 regardless of which axis is selected. Since the radius d1 can be arbitrarily set, it can be set to a smaller value compared to the case of the gizmo. Also, since the rotation axis options C1 to C3 are arranged over the circumference, the width W of the options can also be set wider compared to the case of the gizmo.

[0137] As described above, according to the method for controlling a virtual object according to the present embodiment, it is possible to decrease the value of ID, which is an index of the difficulty until reaching the target in Equation (1), compared to the control method by the gizmo. That is, according to the method for controlling a virtual object according to the present embodiment, it is possible to shorten the operation time until selecting a rotation axis compared to the control method by the gizmo.

[0138] In addition, in the control method using gizmos, since the value of ID in formula (1) varies depending on the axis selected by the user, that is, the difficulty of selection varies depending on the axis, the user may feel stressed when selecting a specific axis. On the other hand, according to the control method of the virtual object according to the present embodiment, the value of ID in formula (1) is the same regardless of which rotation axis is selected. Therefore, according to the control method of the virtual object according to the present embodiment, the operation can be easily executed regardless of which rotation axis is selected.

[0139] <<7. Modification Example>> Next, a modification example according to the present embodiment will be described. In the above, in the first control, an example in which the virtual object rotates in a predetermined direction around the selected rotation axis has been described. Here, in the first control, when the direction in which the virtual object rotates is fixed, the user may feel annoyed when changing the posture of the virtual object to a desired posture. Such a problem will be specifically described with reference to FIG. 11.

[0140] FIG. 11 is a diagram for explaining the problem when the direction in which the virtual object rotates is fixed in the first control. FIG. 11 shows the transition of the posture of the virtual object O5 in the virtual space. Here, it is assumed that the user changes the virtual object O5 in posture (a) to the posture of the virtual object OA to be changed, which is obtained by rotating the virtual object O5 in posture (a) 10 degrees in the negative direction of Pitch.

[0141] Here, in the first control, when the rotation of Pitch is selected, it is assumed that the rotation direction of the virtual object is fixed in the positive direction of Pitch. In this case, after the user selects the rotation of Pitch, the user has to wait for the passage of time until the virtual object O5 rotates from the posture (a) through the postures (b) and (c) to the posture (d) where the virtual object O5 coincides with the posture of the destination virtual object OA. That is, in this case, since the user has to wait for the passage of time until the virtual object O5 rotates 350 degrees in order to rotate the virtual object O5 by 10 degrees, the user may feel annoyed.

[0142] Therefore, hereinafter, a modified example will be described in which the virtual object control unit 222 controls the rotation direction (positive or negative direction of Roll, Pitch, and Yaw) of the virtual object according to the detection result of the gesture of the hand model by the hand model control unit 221. Hereinafter, the positive and negative directions of Roll, Pitch, and Yaw are also referred to as the positive and negative directions of rotation around the X-axis, Y-axis, and Z-axis, respectively. That is, a right-handed screw is directed in the positive direction of each axis, and the rotation direction in which the right-handed screw advances in the positive direction of each axis is defined as the positive direction.

[0143] The hand model control unit 221 according to this modification detects predetermined gestures of the user's left and right hands that the user performs to specify the rotation direction. Hereinafter, such a gesture is also referred to as a "direction-specifying gesture". The direction-specifying gesture may be, for example, an operation of rotating the wrist.

[0144] More specifically, the hand model control unit 221 may detect an operation in which the hand models corresponding to the user's left and right hands are rotated about axes parallel to the X-axis, Y-axis, and Z-axis set with respect to the virtual space, that is, moved in a twisting manner. That is, the hand model control unit 221 may detect an operation in which the hand models corresponding to the user's left and right hands are moved in a twisting manner in the positive and negative directions of Roll, Pitch, or Yaw. In this modification example, an example in which the direction specifying gesture is an operation of twisting the hand model in the positive and negative directions of Roll, Pitch, or Yaw will be described. However, the direction specifying gesture may be another gesture set corresponding to each of the positive and negative directions.

[0145] In the first control, after the rotation axis of the virtual object is selected, when the hand model control unit 221 detects a direction specifying gesture corresponding to the selected rotation axis, the virtual object control unit 222 functions as an acquisition unit that acquires information indicating the rotation direction corresponding to the direction specifying gesture. Then, the virtual object control unit 222 rotates the virtual object in the direction corresponding to the information indicating the rotation direction. The direction specifying gesture corresponding to the selected rotation axis may be an operation of twisting the hand model in the positive or negative direction of Pitch when the rotation axis is the X-axis. Also, when the rotation axis is the Y-axis, it may be an operation of twisting the hand model in the positive or negative direction of Roll. Further, when the rotation axis is the Z-axis, it may be an operation of twisting the hand model in the positive and negative directions of Yaw.

[0146] In the first control, the virtual object control unit 222 may rotate the virtual object according to a direction specifying gesture performed by the hand corresponding to the hand model that the virtual object is not following.

[0147] FIG. 12 is a diagram for explaining an example of a direction-specifying gesture. In the upper part of FIG. 12, a left hand model HL2 and an axis AX3 are shown. The left hand model HL2 is a hand model of the side where the virtual object does not follow in the first control. The axis AX3 represents an axis set in the virtual space.

[0148] In the lower part of FIG. 12, examples of direction-specifying gestures corresponding to each rotation axis are shown. For example, when the selected rotation axis is the X axis, the virtual object control unit 222 determines the rotation direction of the virtual space object according to the direction-specifying gestures of positive and negative (+, -) Rolls, such as twisting the wrist around the X axis, as shown in the lower left of FIG. 12. When the selected rotation axis is the X axis and a positive Roll direction-specifying gesture is detected, the virtual object control unit 222 controls the virtual object to rotate in the positive direction of Roll.

[0149] In FIG. 12, different examples of direction-specifying gestures for Roll, Pitch, and Yaw are shown, but these may be the same gesture. For example, the direction-specifying gesture for Roll may be applied as the direction-specifying gestures for Pitch and Yaw.

[0150] Here, the specific process of controlling the rotation direction of the virtual object by the virtual object control unit 222 will be described. The virtual object control unit 222 determines the rotation speed of the virtual object according to the following formula (2).

[0151] ω rpy = sgn(θ)ω ···(2)

[0152] ω rpyis the rotational speed of the virtual object. θ is the displacement of the angle of the wrist around the rotation axis from when the rotation axis is selected. That is, θ represents the rotation angle by which the wrist has rotated with reference to the posture of the wrist when the rotation axis was selected. For example, when the X-axis is selected as the rotation axis, it is a parameter representing how much the posture of the wrist has rotated around the X-axis when the rotation axis was selected. θ may be detected by the hand model control unit 221 and output to the virtual object control unit 222. ω is the preset rotational speed of the object. ω may be set by the user.

[0153] sgn() is a function that determines positive or negative. sgn(θ) is represented by the following formula (3).

[0154]

Equation

[0155] θ th is a preset threshold value. θ th may be set by the user. As described above, since θ is the displacement around the selected rotation axis from when the rotation axis was selected, it becomes "0" when the wrist is not twisted around the rotation axis after the rotation axis is selected. That is, in this case, sgn(θ) is calculated as "0" according to formula (3). In this case, ω rpy is calculated as "0", so the virtual object control unit 222 does not rotate the virtual object.

[0156] On the other hand, when the wrist is twisted around the selected rotation axis after the rotation axis is selected, θ increases or decreases. More specifically, θ increases when twisted in the positive direction around the rotation axis and decreases when twisted in the negative direction.

[0157] When θ exceeds θ th according to formula (3), sgn(θ) is calculated as "1". Then, ω rpySince it is calculated as “ω”, the virtual object control unit 222 rotates the virtual object at a predetermined rotational speed ω in the positive direction around the rotation axis.

[0158] On the other hand, when θ is less than -θ th sgn(θ) is calculated as “-1” according to Equation (2). Then, ω rpy is calculated as “-ω” according to Equation (2). Therefore, the virtual object control unit 222 rotates the virtual object at a predetermined rotational speed ω in the negative direction around the rotation axis.

[0159] As described above, by recognizing the gesture of twisting the wrist, the rotation direction (positive or negative direction around the rotation axis) can be specified, so that the user can change the posture of the virtual object in a shorter time and more easily without feeling bothered.

[0160] When the rotation axis is selected in the first control, the virtual object control unit 222 may continuously acquire the rotation axis selection gesture until the first control switches to another control method, based on the posture of the wrist when the rotation axis is selected. Then, the virtual object control unit 222 may control to continuously switch the rotation direction based on the comparison between the amount of rotation of the wrist from the reference posture and the threshold value. Thereby, the user can rotate the virtual object in the positive and negative directions to adjust the posture.

[0161] (Operation example according to the modification example) Subsequently, an operation example of the information processing apparatus 10 according to the modification example of the present embodiment will be described. The information processing apparatus 10 according to the present embodiment operates according to the flowchart shown in FIG. 8.

[0162] FIG. 13 is a flowchart showing a subroutine of the process of S105 shown in FIG. 8 in this modification example. The processes of S201 to S210 are the same as the processes described with reference to FIG. 9, and thus the description thereof will be omitted.

[0163] In this modification example, when S203 is "YES" or after the process of S206 is executed, the virtual object control unit 222 determines the rotation direction of the virtual object from the direction specifying gesture detected by the hand model control unit 221 (S217). More specifically, the virtual object control unit 222 determines the positive or negative of the rotation direction of the virtual object. Here, when the direction specifying gesture is not detected (that is, when the wrist is not rotated more than the threshold around the selected axis), it may be determined not to rotate the virtual object.

[0164] Then, the virtual object control unit 222 rotates the virtual object in the determined rotation direction and around the selected rotation axis (S218). Then, the process proceeds to S106. Here, when it is determined in S217 not to rotate the virtual object, the virtual object control unit 222 does not have to rotate the virtual object.

[0165] (Adjustment of rotation speed) So far, a modification example in which the virtual object control unit 222 controls the rotation direction of the virtual object based on the direction specifying gesture performed by the user has been described. Further, the virtual object control unit 222 may function as an acquisition unit that acquires information indicating the rotation speed of the virtual object based on the direction specifying gesture performed by the user. Then, the virtual object control unit 222 rotates the virtual object at a speed corresponding to the information indicating the rotation speed.

[0166] More specifically, the virtual object control unit 222 may control the rotation speed so that the rotation of the virtual object becomes faster as the angle by which the wrist rotates is larger with reference to the posture of the wrist when the rotation axis is selected. More specifically, by adopting the following formula (4) instead of the above formula (3), it becomes possible to adjust the rotation speed.

[0167]

Equation

[0168] θ max is a preset angle. θ max may be set by the user. For example, θ representing the rotation angle of the wrist from the reference posture is θ th is greater than, and θ is θ max When they are equal, ω rpy becomes ω according to Equation (2).

[0169] And, according to Equation (4), the larger the absolute value of θ becomes, that is, the more the wrist is twisted, the larger the absolute value of sgn(θ) becomes. Therefore, according to Equation (2), the more the wrist is twisted, the larger the absolute value of ω rpy becomes. That is, when the virtual object control unit 222 determines the rotation speed of the virtual object based on Equation (2) and Equation (4), the more the user twists the wrist, the faster the rotation of the virtual object is controlled to be.

[0170] Note that the above Equation (4) is an example and is not limited to this as long as it is a conditional expression capable of adjusting the rotation speed. For example, a conditional expression for limiting the maximum and minimum values of sgn(θ) may be added to Equation (4).

[0171] As described above, when the rotation speed of the virtual object can be adjusted by the angle of twisting the wrist, the user can perform a more accurate posture operation of the virtual object. For example, when the user wants to finely adjust the posture of the virtual object, by twisting the wrist slightly to rotate the virtual object slowly, it becomes difficult to miss the timing when the desired posture is achieved.

[0172] <<8. Hardware Configuration Example>> Subsequently, a hardware configuration example of the information processing apparatus 10 according to an embodiment of the present invention will be described.

[0173] Hereinafter, as a hardware configuration example of the information processing apparatus 10 according to an embodiment of the present invention, a hardware configuration example of the information processing apparatus 900 will be described. Note that the hardware configuration example of the information processing apparatus 900 described below is merely an example of the hardware configuration of the information processing apparatus 10. Therefore, the hardware configuration of the information processing apparatus 10 may be obtained by deleting unnecessary configurations from the hardware configuration of the information processing apparatus 900 described below, or new configurations may be added.

[0174] FIG. 14 is a diagram showing a hardware configuration of an information processing apparatus 900 as an example of the information processing apparatus 10 according to an embodiment of the present invention. The information processing apparatus 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0175] The CPU 901 functions as an arithmetic processing unit and a control unit, and controls the overall operation within the information processing apparatus 900 according to various programs. The CPU 901 may be a microprocessor. The ROM 902 stores programs, arithmetic parameters, etc. used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. These are interconnected by a host bus 904 composed of a CPU bus or the like.

[0176] The host bus 904 is connected to an external bus 906 such as a PCI (Peripheral Component Interconnect / Interface) bus via the bridge 905. Note that it is not always necessary to separately configure the host bus 904, the bridge 905, and the external bus 906, and these functions may be implemented on a single bus.

[0177] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, button, microphone, switch, and lever for the user to input information, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. The user who operates the information processing device 900 operates this input device 908, and can input various data to the information processing device 900 and instruct processing operations.

[0178] The output device 909 includes, for example, display devices such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a lamp, and a voice output device such as a speaker.

[0179] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, and a deleting device for deleting data recorded on the storage medium. The storage device 910 is composed of, for example, an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs and various data executed by the CPU 901.

[0180] The communication device 911 is a communication interface composed of, for example, a communication device for connecting to a network. Also, the communication device 911 may support either wireless communication or wired communication.

[0181] The hardware configuration example of the information processing device 10 according to the embodiment of the present invention has been described above.

[0182] <<9. Supplementary Note>> The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that those having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modification examples or correction examples within the scope of the technical idea described in the claims, and it is naturally understood that these also belong to the technical scope of the present invention.

[0183] For example, in the above embodiment, it has been explained that the information processing apparatus 10 can be applied to a scene where work practice in an industrial site is performed, but the present invention is also applicable to the entertainment field such as VR (Virtual Reality) games. In recent years, the development of VR games has been remarkable, and puzzle games that combine virtual objects in a three-dimensional space have also been announced. The present invention is applicable to the entire system that handles such virtual objects.

[0184] Also, in the above embodiment, an example of operating a virtual object in a virtual space has been described, but the virtual space may be superimposed on the real space. For example, the present invention may be applied to technologies such as MR (Mixed Reality) and AR (Augmented Reality).

[0185] Also, in the above embodiment, an example has been described in which a virtual object is operated by a user operating a hand model corresponding to the user's hand arranged in a virtual space, but the hand model does not necessarily have to be arranged. For example, in MR and AR, since an object is superimposed on the real space, by not arranging a hand model in the virtual space, it is possible to express that the virtual object is being operated by the user's own hand in the real space.

[0186] In addition, the hand model arranged in the virtual space does not have to move according to the movement of the user's hand. For example, a controller may be used instead of the user's hand. The controller may include one or more buttons and sensors. The sensor may be, for example, an inertial sensor such as a gyro sensor or an acceleration sensor that can estimate the position in the three-dimensional space and the orientation in the three dimensions based on the sensor data obtained by the sensor. In this case, the hand tracking unit 210 determines the position and orientation of the hand model using the sensor data of the sensors included in the controller instead of the sensor data of the sensor unit 100. When the controller is used, it is sufficient to detect the pressing of a button instead of the hand gesture. For example, it is sufficient to detect the pressing and releasing of the button instead of the opening and closing of the hand.

[0187] In addition, in the above embodiment, an example has been described in which the control in 4DoF including three-directional translational movement and one-directional rotational movement and the control in 6DoF including three-directional translational movement and three-directional rotational movement are switched based on the states of the virtual object and the hand model. However, the number of degrees of freedom and the types of directions in the control method that can be switched based on the states of the virtual object and the hand model are not limited to this, and may be appropriately selected from among three-directional translational movement and three-directional rotational movement. For example, based on the states of the virtual object and the hand model, the control method may transition to a control method that executes only one-directional translational movement. In this case, the moving direction (for example, the X-axis, Y-axis, or Z-axis) may be selected by the same method as the rotation axis UI described above.

Explanation of Reference Numerals

[0188] 10 Information Processing Apparatus 100 Sensor Unit 200 Control Unit 210 Hand Tracking Unit 220 Virtual Space Control Unit 221 Hand Model Control Unit 222 Virtual Object Control Unit 223 State Determination Unit 224 UI Control Unit 225 Image Generation Unit 300 Video Output Unit 400 Memory Unit

Claims

1. An acquisition unit that acquires information indicating a rotation axis of a first virtual object in a virtual space, the rotation axis being selected based on the position of the first virtual object in the virtual space and moved based on the position of an object in real space; A virtual object control unit that rotates the first virtual object around the rotation axis indicated by the information acquired by the acquisition unit within the virtual space; An information processing apparatus comprising the above.

2. The information processing apparatus further comprises a selection control unit that performs control to arrange options for the rotation axis on the same circumference centered on the position corresponding to the first virtual object in the virtual space within the virtual space, The acquisition unit acquires information indicating the rotation axis corresponding to the option when the first virtual object moves to the position of the option on the virtual space. The information processing apparatus according to claim 1.

3. The selection control unit performs control to further arrange in the virtual space a display that guides the rotation direction around the rotation axis corresponding to the option that is closest to the first virtual object in the virtual space among the options for the rotation axis. The information processing apparatus according to claim 2.

4. The virtual object control unit executes a first control to move the position of the first virtual object based on the position of the object while fixing the posture of the first virtual object until the information indicating the rotation axis is acquired by the acquisition unit. The information processing apparatus according to claim 1.

5. The first control further includes control to move the position of the first virtual object based on the position of the object while rotating the first virtual object around the rotation axis when the information indicating the rotation axis is acquired by the acquisition unit. The information processing apparatus according to claim 4.

6. The virtual object control unit controls to arrange a second virtual object corresponding to the object in the virtual space, The information processing apparatus, further comprises a determination unit that determines whether the states of the first virtual object and the second virtual object are in a first state or a second state, The virtual object control unit executes the first control when the determination unit determines that the state is the first state. The information processing apparatus according to claim 5.

7. When the determination unit determines that the states of the first virtual object and the second virtual object are in the second state, the virtual object control unit controls the position and orientation of the first virtual object according to the position and orientation of the target in the real space. The information processing apparatus according to claim 6.

8. The determination unit determines whether the state is the first state or the second state based on whether the first virtual object and the second virtual object are in contact with each other in the virtual space. The information processing apparatus according to claim 7.

9. The target is the left and right hands of the user. The second virtual object includes a left hand object corresponding to the left hand of the user and a right hand object corresponding to the left hand of the user. When the first virtual object is in contact with the left hand object and the right hand object, the determination unit determines that the state is the first state. The information processing apparatus according to claim 8.

10. When the first virtual object is in contact with only one of the left hand object and the right hand object, the determination unit determines that the state is the second state. The information processing apparatus according to claim 9.

11. When the first virtual object is not in contact with the left hand object and the right hand object, the determination unit determines that the state is the third state. When the determination unit determines that the state is the third state, the virtual object control unit executes control to fix the position and orientation of the first virtual object. The information processing apparatus according to claim 9.

12. The acquisition unit further acquires information indicating the rotation direction around the rotation axis. The virtual object control unit rotates the first virtual object in the rotation direction indicated by the information acquired by the acquisition unit. The information processing apparatus according to claim 1.

13. The acquisition unit acquires information indicating the rotation direction based on the rotation direction of the target in the real space. The information processing apparatus according to claim 12.

14. The acquisition unit further acquires information indicating the rotation speed around the rotation axis. The virtual object control unit rotates the first virtual object at the rotation speed indicated by the information acquired by the acquisition unit. The information processing apparatus according to claim 1.

15. The acquisition unit acquires information indicating the rotation speed based on the rotation angle from a reference on the real space of the target, the information processing apparatus according to claim 14.

16. acquiring information indicating the rotation axis of the first virtual object, which is selected based on the position of the first virtual object in the virtual space and is moved based on the position of the target in the real space; rotating the first virtual object around the rotation axis indicated by the acquired information within the virtual space; An information processing method executed by a computer, including:

17. an acquisition unit that acquires information indicating the rotation axis of the first virtual object, which is selected based on the position of the first virtual object in the virtual space and is moved based on the position of the target in the real space; a virtual object control unit that rotates the first virtual object around the rotation axis indicated by the information acquired by the acquisition unit within the virtual space; A program that causes a computer to function as:

Images