Information processing apparatus, method for controlling information processing apparatus, information processing system, and program

Abstract

To provide an information processing apparatus capable of accurately and easily adjusting a position of a virtual object at a destination thereof when the virtual object is moved in a mixed reality space, a method for controlling the information processing apparatus, an information processing system, and a program.SOLUTION: In a functional configuration having an HMD 150 and a compact operation device 160, the HMD serving as an information processing apparatus includes: a drawing unit 105 serving as generation means that generates a mixed reality space; a control unit 107 serving as switching means that switches a moving operation of the compact operation device which is operation means that performs moving operation to move a virtual object in a mixed reality space, between a first moving operation which allows the virtual object to be continuously moved, and a second moving operation which allows the virtual object to be moved stepwise; and a CG control unit 104 serving as direction restricting means which restricts directions in which the virtual object is movable in the second moving operation, to at least one direction.SELECTED DRAWING: Figure 1B

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, an information processing system, and a program. [Background technology]

[0002] In recent years, devices have become known that allow users to experience mixed reality by overlaying information such as images from a virtual space on images from a real space in real time. These devices can display a composite image in which a virtual space image, such as a CG image, is superimposed on an image of the entire or partial real space captured by an imaging device such as a video camera. Furthermore, these devices allow users to manipulate the CG and move it as desired. For example, with Microsoft® HoloLens 2, when a user manipulates a CG with their hand, a linear image called a "ray" is displayed between the hand and the CG, allowing the user to clearly identify the CG being manipulated. This allows for easy manipulation of the CG being manipulated. For example, Patent Document 1 discloses a technology that displays a CG (virtual object) as a manipulation axis within a composite space that combines real and virtual spaces, when the CG is moved from a source to a destination. The technology described in Patent Document 1 allows the CG to be moved along the manipulation axis. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-129167 Summary of the Invention [Problem to be solved by the invention]

[0004] However, although the technology described in Patent Document 1 makes it possible to move CG along an operation axis within a composite space, it may be difficult to precisely adjust the position of the CG at the destination, i.e., to fine-tune the placement position. Also, even when moving the CG to be moved using a ray displayed between the user's hand and the CG, if the movement of the user's hand becomes unstable due to, for example, camera shake, it may be difficult to precisely adjust the position of the CG at the destination.

[0005] The present invention has been made in view of the above-mentioned problems, and aims to provide an information processing device, a control method for the information processing device, an information processing system, and a program that are capable of accurately and easily adjusting the position of a virtual object at a destination when the virtual object is moved in a virtual reality space or a mixed reality space. [Means for solving the problem]

[0006] In order to achieve the above object, the information processing device of the present invention is characterized by comprising: an operation means for performing a movement operation to move a virtual object within a virtual reality space or a mixed reality space; a switching means for switching the movement operation by the operation means between a first movement operation that enables continuous movement of the virtual object and a second movement operation that enables stepwise movement of the virtual object; and a direction restriction means for restricting the direction in which the virtual object can be moved by the second movement operation to at least one direction. [Effects of the Invention]

[0007] According to the present invention, when a virtual object is moved in a mixed reality space, the position of the virtual object at the destination can be adjusted accurately and easily. [Brief explanation of the drawings]

[0008] [Figure 1A]1 is a block diagram showing an example of a hardware configuration when the information processing device according to the first embodiment is applied to an HMD (Head Mounted Display). [Figure 1B] FIG. 2 is a block diagram showing an example of the functional configuration of an HMD and a small-sized operating device. [Figure 2] FIG. 1 is a diagram showing a state in which an HMD is worn on a user's head and a small operating device is worn on the user's finger. [Figure 3] FIG. 10 is a diagram illustrating the relationship between a user's hand, a virtual object, and a ray. [Figure 4] FIG. 10 is a diagram showing a data list for determining a movement range of a virtual object. [Figure 5] FIG. 10 is a diagram showing an example of an image displayed on a display of an HMD. [Figure 6] 10 is a flowchart showing processing executed by the HMD. [Figure 7] FIG. 10 is a diagram illustrating an example of a state in which a virtual object is moved within a mixed reality space displayed on an HMD. [Figure 8] FIG. 10 is a diagram showing a state in which a small operating device according to a second embodiment is worn on a user's finger. [Figure 9] FIG. 10 is a diagram showing an example of an image displayed on a display of an HMD. DETAILED DESCRIPTION OF THE INVENTION

[0009] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in each embodiment. For example, each component constituting the present invention can be replaced with any configuration that can perform the same function. Also, any component may be added. Furthermore, any two or more configurations (features) of each embodiment can be combined.

[0010] First Embodiment A first embodiment will be described below with reference to FIGS. 1A to 8. FIG. 1A is a block diagram showing an example of a hardware configuration when an information processing device according to the first embodiment is applied to a part of an HMD (Head Mounted Display). In this embodiment, the HMD 150 shown in FIG. 1A is a device worn by a user on the head. Note that devices to which the information processing device can be applied are not limited to the HMD 150 and may include, for example, desktop or notebook personal computers, tablet terminals, smartphones, etc. As shown in FIG. 1A, the HMD 150 includes a CPU 801, a RAM 802, a ROM 803, a display 103, and a camera 106. The CPU 801, together with the ROM 803 and the RAM 802, constitutes a control unit 107 (see FIG. 1B) that controls the overall operation of the HMD 150. The ROM 803 is a storage medium that stores various programs and operating conditions for the above-mentioned operations. The programs include programs for causing the CPU 801 (computer) to execute each part and each means (control method of the information processing device) of the HMD 150. The RAM 802 is also used as a work memory for storing temporary data for processing executed by the CPU 801.

[0011] The display 103 is a display means for displaying a mixed reality (MR) space in which an image of real space and an image of virtual space are combined, and is configured with, for example, a liquid crystal display. The user can view the mixed reality space displayed on the display 103 in a state in which the HMD 150 is worn on the user's head (hereinafter referred to as the "wearing state"). The camera 106 is an imaging means capable of capturing an image of the three-dimensional space in front of the camera 106 in the wearing state, i.e., the real space, and has a camera fixed to the housing of the HMD 150 corresponding to the user's left eye and a camera corresponding to the user's right eye. This makes it possible to capture an image of the real space in stereoscopic vision. The display 103 may also be a display means for displaying a virtual reality (VR) space.

[0012] FIG. 1B is a block diagram showing an example of the functional configuration of an HMD and a small operating device. As shown in FIG. 1B, the HMD 150 includes an image storage unit 101, a ray control unit 102, a display 103, a CG control unit 104, a drawing unit (generation means) 105, a camera 106, a control unit 107, and a communication unit 108. The image storage unit 101 stores images captured by the camera 106, i.e., images of real space. The ray control unit 102 controls the position and orientation of a ray. A "ray" is a linear image displayed between a virtual object (CG model; virtual object) described below when a user moves the virtual object with their hand. This ray clarifies the virtual object to be moved. The ray control unit 102 detects the position and orientation of the user's hand based on the captured image stored in the image storage unit 101. The ray control unit 102 can then determine (calculate) the start and end positions of the ray based on the detected position and orientation of the hand.

[0013] The CG control unit 104 manages information about virtual objects included in images of the virtual space. The virtual objects are displayed on the display 103 using CG (Computer Graphics). The CG control unit 104 determines the positions of virtual objects within the mixed space and controls the selection and movement of the virtual objects. The drawing unit 105 generates a mixed reality space by combining an image of the real space with an image of the virtual space including the virtual objects (generation process). This mixed reality space is displayed on the display 103. The drawing unit 105 can also draw virtual objects and the like at positions determined by the CG control unit 104 on the captured images stored in the image storage unit 101, and can also draw and combine rays determined by the ray control unit 102. The communication unit 108 is composed of, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller, and can communicate with the small operating device 160. The communication unit 108 realizes short-distance wireless communication in accordance with the IEEE802.15 standard (so-called Bluetooth (registered trademark)) by outputting the weighted wireless signal from the antenna and demodulating the wireless signal received by the antenna. In this embodiment, the Bluetooth communication employs version 5.1 of Bluetooth Low Energy, which has low power consumption.

[0014] The HMD 150 configured as described above is communicatively connected to the small-sized operating device 160. The small-sized operating device 160 is a controller detachably attached to the finger of a user using the HMD 150, and functions as an operating means for performing a movement operation to move a virtual object in a mixed reality space (operation process). The control unit 107 of the HMD 150 can switch the movement operation performed by the small-sized operating device 160 between a first movement operation and a second movement operation (switching process). The "first movement operation" is an operation that enables continuous movement of a virtual object. The "second movement operation" is an operation that enables gradual (discrete) movement of a virtual object. In this manner, in this embodiment, the control unit 107 functions as a switching means that switches the movement operation performed by the small-sized operating device 160 between the first movement operation and the second movement operation. Note that in the HMD 150, a section that functions as the switching means may be provided separately from the control unit 107. The instruction to switch between the first movement operation and the second movement operation (switching operation) is given to the control unit 107 by the small-sized operating device 160. Furthermore, the small operating device 160 is not limited to being detachably attached to the user's finger, but may be configured to be at least grippable by the user's hand.

[0015] As shown in FIG. 1B , the small-sized operating device 160 includes a communication unit 109, an OTP (Optical Tracking Pad) 110, and an orientation detection unit 111. The communication unit 109 has a configuration similar to that of the communication unit 108. The communication unit 109 transmits position information of the user's fingertip detected by the OTP 110 and orientation information regarding the orientation of the small-sized operating device 160 detected by the orientation detection unit 111 to the HMD 150. The user can perform a sliding operation (touch operation) with their fingertip on the OTP 110. The OTP 110 can detect the position information of the user's fingertip by receiving this sliding operation. The orientation detection unit 111 is an inertial measurement unit (IMU) that detects translational motion in three orthogonal axial directions and rotational motion around each axis. The orientation detection unit 111 includes an acceleration sensor that detects translational motion and an angular velocity sensor (gyro sensor) that detects rotational motion. This allows orientation information regarding the orientation of the small-sized operating device 160 to be detected.

[0016] FIG. 2 illustrates a state in which an HMD is worn on a user's head and a small operating device is worn on the user's finger. As shown in FIG. 2(a), when the HMD is worn, the camera 106 can capture an image of the real space including the user's hand 201. A small operating device 160 is worn on the index finger of the hand 201. FIG. 2(b) is an enlarged perspective view of the small operating device in FIG. 2(a). As shown in FIG. 2(b), the small operating device 160 is ring-shaped and detachably worn on the finger, and the OTP 110 is disposed on its outer periphery. For example, while wearing the small operating device 160 on the index finger, the user can slide the OTP 110 with the tip of their thumb. In this embodiment, the CG control unit 104 can acquire position information of the fingertip on the OTP 110 over time via the communication units 108 and 109. Then, based on changes in this position information over time, the user's gesture, i.e., instructions from the user, can be determined. This allows virtual objects to be operated in a mixed reality space. Fig. 2(c) is a diagram showing a state in which the small-sized operating device 160 worn on the user's finger is oriented in the direction of arrow 202, i.e., horizontally. Fig. 2(d) is a diagram showing a state in which the small-sized operating device 160 worn on the user's finger is oriented in the direction of arrow 203, i.e., vertically. The orientation detection unit 111 can detect whether the small-sized operating device 160 is in the state of Fig. 2(c) or the state of Fig. 2(d), for example.

[0017] FIG. 3 is a diagram illustrating the relationship between a user's hand, a virtual object, and a ray. An image 300 shown in FIG. 3 can be an example of an image displayed on the display 103 of the HMD 150. The image 300 includes a user's hand 201 and a virtual object 301. In the HMD 150, the third joint of the index finger of the user's hand 201 is defined as the origin O305, the left-right direction (horizontal direction) as the x-axis direction, the up-down direction (vertical direction) as the y-axis direction, and the front-back direction (depth direction) as the z-axis direction. A ray 321 is represented by a straight arrow starting from the origin O305 and ending at the center point O306 of the virtual object 301. Therefore, the ray 321 can be displayed by detecting the coordinates of the origin O305 and the center point O306. In this embodiment, the virtual object 301 is, for example, a sphere centered at the center point O306. The shape of the virtual object 301 can be displayed by detecting the coordinates of multiple points equidistant from the center point O306. Note that the shape of the virtual object 301 can also be represented by, for example, a point cloud having a plurality of arbitrary feature points. Also, by regarding the position of the small-sized operating device 160 as the origin 305, it is possible to display a ray extending from the small-sized operating device 160. The direction of the ray is determined according to the orientation or posture of the small-sized operating device 160. In this embodiment, the origin 305 is set at the third joint of the index finger of the hand 201, but is not limited thereto and may be set at the center position of the hand 201, for example.

[0018] FIG. 4 is a diagram showing a data list for determining the movement range of a virtual object. FIG. 4(a) shows a CG virtual object, the center position of the virtual object, and a point cloud (shape point cloud) representing the shape of the virtual object. This information is managed by, for example, an application stored in the ROM 803 of the HMD 150. As shown in FIG. 4(a), for example, virtual object 1 (CG1) has a center position of (x1, y1, z1), and the positions of each point included in the shape point cloud are (x11, y11, z11), (x12, y12, z12), etc. This makes it possible to determine (specify) the position and shape of CG1. When virtual object 1 is moved, the coordinates of the center and the coordinates of each point included in the shape point cloud are updated. Note that the number of points included in the shape point cloud varies depending on the shape of the virtual object. Furthermore, the method for determining the shape of a virtual object is not limited to using coordinates; any information that at least contributes to determining the shape of the virtual object may be used. FIG. 4(b) shows the relationship between the start position (start point) and end position (end point) of the ray. The start position (xLS, yLS, zLS) of the ray is, for example, the third joint of the index finger of the user's hand 201 in real space. The end position (xLE, yLE, zLE) of the ray can be specified, for example, by a predetermined operation by the user. This predetermined operation is not particularly limited, and examples include a gesture or the user sliding their fingertip on the OTP 110 of the small-sized operating device 160. Note that the predetermined operation may be performed using, for example, a device with a touchpad, a device with mechanical buttons, a joystick, or the like, in addition to the small-sized operating device 160. When the user's hand moves, the start position (xLS, yLS, zLS) and end position (xLE, yLE, zLE) of the ray are updated. Note that the coordinates in real space corresponding to each of the coordinates shown in FIG. 4(a) can be calculated by adding the start position of the ray shown in FIG. 4(b).

[0019] FIG. 5 is a diagram showing an example of an image displayed on the display of the HMD. Image 500 shown in FIG. 5 is an image of a mixed reality space. Image 500 includes virtual objects 511, 512, 513, and 514. Virtual object 511 is a table and corresponds to virtual object 1 in FIG. 4(a). Virtual object 512 is a sofa and corresponds to virtual object 2 in FIG. 4(a). Virtual object 513 is a television and corresponds to virtual object 3 in FIG. 4(a). Virtual object 514 is a potted plant and corresponds to virtual object 4 in FIG. 4(a). Image 500 also includes a ray 510, a user's hand 201 captured by camera 106, and a small operating device 16 captured by camera 106 and worn near the third joint of the index finger of hand 201. In FIG. 5 , the ray 510 starts from the third joint of the index finger of the hand 201, i.e., the small-sized operating device 16, and ends at the center of the virtual object 511. By moving the hand 201 together with the small-sized operating device 16 in this state, the ray 510 follows the movement of the hand 201, and the virtual object 511 can be moved to any position. The virtual object 511 can also be moved to any position by, for example, the user sliding their thumb or the like on the OTP 110 of the small-sized operating device 160 in the direction of the desired position. In this manner, in this embodiment, the small-sized operating device 16 functions as an operating unit for performing a movement operation to move a virtual object within a mixed reality space. The image 500 also includes a window 521, a floor 522, a wall 523, and a ceiling 524. These are images of real space captured by the camera 106.

[0020] Fig. 6 is a flowchart showing processing executed by the HMD. A program based on the flowchart shown in Fig. 6 is started in a state in which the HMD 150 and the small operating device 160 are activated. As shown in Fig. 6, in step S601, the control unit 107 of the HMD 150 controls the camera 106 to acquire a captured image (stereo camera image) showing the user's hands, i.e., an image of real space, and the process proceeds to processing step S602. This captured image is stored in the image storage unit 101.

[0021] In step S602, the control unit 107 controls the ray control unit 102 to detect (extract) the user's hand included in the captured image stored in step S601, and the process proceeds to step S603. The method for detecting the user's hand is not particularly limited, and for example, an image recognition method or the like can be used. The method for acquiring the distance from the HMD 150 to the user's hand in real space is not particularly limited, and for example, a stereo image method or the like can be used. Then, based on the detection of the user's hand and the distance from the HMD 150 to the user's hand in real space, the position of the third joint of the index finger, which is position information of the user's hand in the captured image, is acquired. The position information of the user's hand may be acquired using, for example, a distance sensor or the like.

[0022] In step S603, the control unit 107 controls the communication unit 108 to receive, from the small-sized operation device 160, position information of the touch of the user on the OTP 110 of the small-sized operation device 160, and the process proceeds to step S604. The position information received in step S603 is sequentially stored in the RAM 802.

[0023] In step S604, the control unit 107 controls the ray control unit 102 to calculate the position of the start point of the ray, and the process proceeds to step S605. In this embodiment, the start point of the ray is set to the position of the third joint of the index finger acquired in step S602.

[0024] In step S605, the control unit 107 controls the ray control unit 102 to determine the end position (absolute position) of the ray, and the process proceeds to step S606. The method for determining the end position of the ray is not particularly limited. For example, the ray control unit 102 first determines the direction of the hand (or the direction of the user's middle finger, etc.) obtained from the captured image as the direction of the ray. Next, the ray control unit 102 determines the position that is the ray length away from the start position of the ray in the ray direction as the end position of the ray. Note that the length of the ray can also be specified by, for example, a predetermined operation (gesture) by the user. This predetermined operation is not particularly limited, and an example of this is a sliding operation of the user's fingertip on the OTP 110. In this case, the more sliding operations are performed, the longer the length of the ray. The length of the ray may also be a fixed length. The direction of the ray may also be the direction of a sliding operation of the fingertip on the OTP 110. In this case, the ray control unit 102 sets the direction of the ray to the direction of the sliding operation of the fingertip on the OTP 110 based on the position information stored in the RAM 802 in step S603. The final end position of the ray may be determined by adding the movement direction and movement amount (movement direction instruction information) determined based on the hand position information to the end position of the ray determined based on the hand position information. The start position (start point) and end position (end point) of the ray in FIG. 4(b) are updated each time each point is determined.

[0025] In step S606, the control unit 107 controls the CG control unit 104 to determine whether a virtual object, which is CG, has been selected by the user and is being moved by a first movement operation. If the determination in step S606 determines that the virtual object is being moved, the process proceeds to step S608. On the other hand, if the determination in step S606 determines that the virtual object is not being moved, the process proceeds to step S607. Here, "a state in which a virtual object is moving" refers to a state in which a virtual object has been selected by the user and the selection of the virtual object has not been deselected.

[0026] In step S607, the control unit 107 controls the CG control unit 104 to control the virtual object in accordance with the user's hand gesture, and the process proceeds to step S614. Specifically, the CG control unit 104 monitors the user's hand movement (gesture) based on the captured image and controls the virtual object in accordance with the hand movement. For example, the CG control unit 104 monitors the captured image, and when it determines that the user has made a gesture such as lightly tapping the air with their index finger to select a virtual object, the CG control unit 104 selects the virtual object located at the end position of the ray. Note that the captured image used in step S607 may be a single still image, or may be a plurality of still images acquired consecutively from a predetermined time before the current time to the current time.

[0027] In step S608, the control unit 107 controls the CG control unit 104 to determine whether a transition has occurred from the first movement operation to the second movement operation, i.e., whether a switch has occurred. This determination is made, for example, based on whether the end point of the ray after the virtual object has moved by the first movement operation remains stationary for a predetermined period of time. Therefore, if the end point of the ray remains stationary for the predetermined period of time, it is determined that a transition has occurred to the second movement operation. Note that the predetermined period of time used for this determination is preferably settable and changeable as appropriate. Furthermore, the determination in step S608 may be made based on whether or not the end point position of the ray has changed after the virtual object has moved by the first movement operation. Therefore, if the change in the end point position of the ray is within a predetermined range for a predetermined period of time, it is determined that a transition has occurred to the second movement operation. Note that the predetermined period of time and the predetermined range used for this determination are also preferably settable and changeable as appropriate. Furthermore, the determination in step S608 may be made based on whether or not a touch operation has been performed on the OTP 110 of the small-sized operating device 160. Therefore, if a touch operation has been performed, it is determined that a transition has occurred to the second movement operation. Furthermore, information indicating that the operation has transitioned to the second movement operation is held in RAM 802, and this held state is maintained until the operation returns to the first movement operation. If the result of the determination in step S608 indicates that the operation has transitioned to the second movement operation, the process proceeds to step S609. On the other hand, if the result of the determination in step S608 indicates that the operation has not transitioned to the second movement operation, the process proceeds to step S612.

[0028] In step S609, the control unit 107 controls the communication unit 108 to acquire orientation information of the small-sized operating device 160 from the orientation detection unit 111 of the small-sized operating device 160, and the process proceeds to processing step S610.

[0029] In step S610, the control unit 107 controls the CG control unit 104 to determine the direction in which the virtual object can be moved by the second movement operation, based on the orientation information of the small-sized operating device 160 acquired in step S609 (direction restriction process). In this embodiment, the CG control unit 104 thus functions as a direction restriction unit that restricts the direction in which the virtual object can be moved by the second movement operation. The directions determined in step S610 are two intersecting directions, i.e., two-dimensional movement directions. For example, as shown in FIG. 2(c), when the small-sized operating device 160 is in the direction of arrow 202, i.e., in a state where the small-sized operating device 160 is oriented horizontally, the two-dimensional movement directions in the mixed reality space 2000 are determined to be the x-axis direction (horizontal direction) and the z-axis direction (depth direction). Furthermore, as shown in FIG. 2(d), when the small-sized operating device 160 is in the direction of arrow 203, i.e., in a state where the small-sized operating device is oriented vertically, the two-dimensional movement directions in the mixed reality space 2000 are determined to be the x-axis direction (horizontal direction) and the y-axis direction (vertical direction). The combinations of the x-axis, y-axis, and z-axis directions are not limited to these, and are changed depending on the direction (posture) of the small-sized operating device 160.

[0030] Furthermore, in step S610, the control unit 107 controls the CG control unit 104 to determine the movement amount M by which the virtual object can be moved by the second movement operation. The "movement amount M" is the maximum distance that can be moved in each step of the second movement operation, which allows the virtual object to move in stages. This movement amount M is set to be shorter than one continuous movement distance that can be moved by the first movement operation. Note that the movement amount M may be maintained at a constant value or may be changed depending on the size of the virtual object. When the movement amount M is changed depending on the size of the virtual object, the larger the virtual object, the larger the movement amount M, and the smaller the virtual object, the smaller the movement amount M. After step S610 is executed, the process proceeds to step S611.

[0031] In step S611, the control unit 107 controls the CG control unit 104 to determine whether or not to move the virtual object by the second movement operation. This determination is made based on the movement direction instruction information used in step S605. If the determination in step S611 determines that the virtual object will be moved by the second movement operation, the process proceeds to step S613. On the other hand, if the determination in step S611 determines that the virtual object will not be moved by the second movement operation, the process proceeds to step S614.

[0032] In step S612, the control unit 107 controls the CG control unit 104 to update the end position in Fig. 4(b), and the process proceeds to step S614. Specifically, the CG control unit 104 updates the end position of the virtual object so that the end position of the ray calculated in step S605 becomes the same position as the coordinates converted into the coordinate system of the mixed reality space.

[0033] In step S613, the control unit 107 controls the CG control unit 104 to update the position in Fig. 4(a), and the process proceeds to step S614. Specifically, the CG control unit 104 updates the position of the virtual object by adding a certain amount of movement to the movement direction indicated by the movement direction instruction information used in step S605. For example, if the position of the virtual object is (x1, y1, z1) and the two-dimensional movement direction is the x-axis and z-axis, and the movement direction instruction information indicates movement forward, the CG control unit 104 adds the amount of movement M so that the position of the virtual object becomes (x1, y1, z1 + M).

[0034] In step S614, the control unit 107 controls the drawing unit 105 to draw the virtual object, and the process proceeds to step S615. The drawing arrangement of the virtual object can be the position updated in step S612 or step S613.

[0035] In step S615, the control unit 107 controls the CG control unit 104 to determine whether or not to place the virtual object selected by the user. For example, if the CG control unit 104 determines that the user has extended all of his or her fingers and made a gesture as if to place a virtual object, the control unit 107 determines that the virtual object is to be placed. If the determination in step S615 determines that the virtual object is to be placed, the process proceeds to step S617. On the other hand, if the determination in step S615 determines that the virtual object is not to be placed, the process proceeds to step S616.

[0036] In step S616, the control unit 107 controls the drawing unit 105 to draw the ray, and the process proceeds to step S618. The drawing arrangement of the ray is obtained from the data list shown in FIG.

[0037] In step S617, the control unit 107 clears the information that the virtual object has been selected, the information related to the second movement operation state, and the start and end positions of the ray, and the process proceeds to step S620.

[0038] In step S618, the control unit 107 determines whether or not the state is the second movement operation state, based on the information indicating a transition to the second movement operation stored in the RAM 802 in step S608. If the determination in step S618 determines that the state is the second movement operation state, the process proceeds to step S619. On the other hand, if the determination in step S618 determines that the state is not the second movement operation state, the process proceeds to step S620.

[0039] In step S619, the control unit 107 controls the drawing unit 105 to draw a scale, which will be described later, and the process proceeds to processing step S620.

[0040] In step S620, the control unit 107 displays the image in which the virtual object and ray are drawn on the display 103 as an image in mixed reality space, and the process proceeds to step S619.

[0041] In step S621, the control unit 107 determines whether or not to terminate the execution of the program based on the flowchart shown in Fig. 6. For example, the control unit 107 determines to terminate the execution of the program when it receives an instruction from the user to terminate the execution of the program. If the determination in step S621 determines that the execution of the program is to be terminated, the process ends. On the other hand, if the determination in step S621 determines that the execution of the program is not to be terminated, the process returns to step S601, and the subsequent steps are executed sequentially.

[0042] 7A and 7B are diagrams showing an example of a state in which a virtual object is moved in a mixed reality space displayed on an HMD. Fig. 7A shows a state in which a virtual object 511 to be moved is selected by a ray 510 extending from a user's hand 201. In Fig. 7A, virtual objects 511, 512, and 513 are arranged in this order in a straight line with a gap between them. The state shown in Fig. 7A transitions to the state shown in Fig. 7B.

[0043] 7(b) shows a state in which virtual object 511 is moved together with ray 510 by a first movement operation by further moving hand 201, and is placed between virtual object 512 and virtual object 513. In this way, the first movement operation is used to move virtual object 511 from a position in front of virtual object 512, which is its source, to a position between virtual object 512, which is its destination, and virtual object 513. A transition is made from the state shown in FIG. 7(b) to the state shown in FIG. 7(c).

[0044] FIG. 7(c) shows a state in which the second movement operation has been switched to and the virtual object 511 is being placed at an arbitrary position between the virtual objects 512 and 513 by moving the hand 201. In this way, the second movement operation is used to adjust the position of the virtual object 511 at the destination. In a second movement operation enabled state in which the second movement operation is possible, an x-axis 701 (horizontal axis) and a z-axis 702 (depth axis) are displayed with the center point 703 of the virtual object 511 to be moved as the origin. In this case, as described above, the small operating device 160 is oriented in the direction of the arrow 202, that is, horizontally (see FIG. 2(c)). The x-axis 701 and the z-axis 702 are displayed as straight lines that indicate the movement direction of the virtual object 511 restricted by the second movement operation. Note that the x-axis 701 and the z-axis 702 are preferably displayed as straight lines, for example, emphasized as thick lines. This allows the user to grasp the directions in which the virtual object 511 can be moved. The y-axis 704 may also be displayed, but it is preferable that the y-axis 704 be less emphasized than the x-axis 701 and z-axis 702, for example, by using a thin line or a pale color. This makes it possible to distinguish between directions in which the virtual object 511 can be moved by the second movement operation and other directions. Furthermore, in the second movement operation possible state, scales indicating the movement amount M (maximum distance) are displayed on the x-axis 701 and z-axis 702. The virtual object 511 can be moved by one scale at a time by the second movement operation. A transition is made from the state shown in FIG. 7(c) to the state shown in FIG. 7(d).

[0045] FIG. 7(d) shows a state in which the virtual object 511 is moved by one graduation along the z-axis 702 (one direction) of the x-axis 701 and z-axis 702 by performing a sliding operation on the OTP 110 of the small-sized operating device 160 in a predetermined direction. At this time, movement along the x-axis 701 (the other direction) is restricted. For example, movement along the x-axis 701 is not possible, that is, control is performed to prohibit movement along the x-axis 701. As a result, even if the sliding operation on the OTP 110 of the small-sized operating device 160 is slightly deviated from the predetermined direction, the virtual object 511 can be moved smoothly along the z-axis 702. A transition is made from the state shown in FIG. 7(d) to the state shown in FIG. 7(e).

[0046] FIG. 7(e) shows a state in which the second movement operation possible state is maintained and the position of the virtual object 511 is being further adjusted by moving the hand 201. In this state, an x-axis 701 (horizontal axis) and a y-axis 704 (depth axis) are displayed with the center point 703 of the virtual object 511 as the origin. In this case, as described above, the small operating device 160 is oriented in the direction of the arrow 203, i.e., vertically (see FIG. 2(d)). The x-axis 701 and the y-axis 704 are displayed as straight lines indicating the movement direction of the virtual object 511 restricted by the second movement operation. Note that the x-axis 701 and the y-axis 704 are preferably displayed as straight lines, for example, emphasized as thick lines. This allows the user to grasp the directions in which the virtual object 511 can be moved. Furthermore, in this second movement operation possible state, scales indicating the movement amount M are displayed on the x-axis 701 and the y-axis 704. The virtual object 511 can be moved by one scale at a time by the second movement operation. The state shown in FIG. 7(e) transitions to the state shown in FIG. 7(f).

[0047] Fig. 7(f) shows a state in which the virtual object 511 is moved by one scale along the y-axis 704 of the x-axis 701 and y-axis 704 by performing a sliding operation on the OTP 110 of the small-sized operating device 160 in one predetermined direction. At this time, the movement along the x-axis 701 is restricted. A transition occurs from the state shown in Fig. 7(f) to the state shown in Fig. 7(g).

[0048] 7(g) shows a state in which the x-axis 701, the y-axis 704, etc. have disappeared and the placement of the virtual object 511 has been completed. The instruction to complete the placement of the virtual object 511 is given, for example, by the user sliding or tapping the OTP with their fingertip. In this way, the OTP also functions as an operation means that allows an operation to determine the completion of the placement (movement) of the virtual object 511. This makes it possible to prevent the virtual object 511 from moving unintentionally within the image 500.

[0049] As described above, by using the HMD 150 and the small-sized operating device 160, it is possible to quickly move the virtual object 511 from the origin to the destination in the image 500, which is an image of the mixed reality space, by a first movement operation. Then, at the destination, it is possible to accurately and easily adjust the position of the virtual object 511 by a second movement operation. Note that instead of the small-sized operating device 160, the user's hand 201 displayed on the image 500 may be used as the operating means.

[0050] Note that the operation on the virtual object 511 is not limited to a movement operation in the x-axis, y-axis, and z-axis directions, and may be, for example, a rotation operation around each axis, i.e., a rotation operation in the roll direction, pitch direction, and yaw direction. In this case, for example, at the position where the virtual object 511 is placed, a first movement operation can be used to continuously rotate the virtual object 511 in accordance with a gesture, and a second movement operation can be used to rotate the virtual object 511 stepwise in a predetermined rotation direction. Any angle may be set as the stepwise angle, such as rotating by 10 degrees. Furthermore, the first movement operation may be used to continuously move the virtual object 511 in the x-axis, y-axis, and z-axis directions, and the second movement operation may be used to rotate the virtual object stepwise in a predetermined rotation direction. In this way, the first movement operation and the second movement operation may have different functions as the axis serving as the operation reference (movement along an axis in the first movement operation, and rotation around an axis in the second movement operation).

[0051] Second Embodiment The second embodiment will be described below with reference to FIGS. 8 and 9. Differences from the previous embodiment will be mainly described, and similar points will not be described again. FIG. 8 is a diagram showing a state in which a small-sized operating device according to the second embodiment is worn on a user's finger. FIG. 8(a) is a diagram showing a state in which the small-sized operating device 160 worn on the user's finger is oriented in the direction of arrow 202, i.e., horizontally, similar to FIG. 2(c). FIG. 8(b) is a diagram showing a state in which the small-sized operating device 160 worn on the user's finger is oriented in the direction of arrow 203, i.e., vertically, similar to FIG. 2(d). FIG. 8(c) is a diagram showing a state in which the small-sized operating device 160 worn on the user's finger is oriented in the direction of arrow 901, i.e., diagonally upward to the right. The orientation detection unit 111 can detect which of the states shown in FIGS. 8(a) to 8(c) the small-sized operating device 160 is in, for example.

[0052] 9A and 9B are diagrams illustrating an example of an image displayed on the display of the HMD. FIG. 9A illustrates a state in which the position of the virtual object 511 is being adjusted by moving the hand 201 in the second movement operation enabled state. In this state, an x-axis 701 (a horizontal axis) with the center point 703 of the virtual object 511 as its origin and a diagonal axis 1001 (an axis pointing diagonally upward to the right) are displayed. The diagonal axis 1001 is an intermediate axis between the y-axis and the z-axis, i.e., an axis extending in a direction inclined with respect to the y-axis (or x-axis). When the x-axis 701 and the diagonal axis 1001 are displayed, the small operating device 160 is pointing in the direction of the arrow 910, i.e., diagonally upward to the right (see FIG. 8C). The x-axis 701 and the diagonal axis 1001 are displayed as straight lines indicating the movement direction of the virtual object 511 restricted by the second movement operation. In addition, in this second movement operation enabled state, scales indicating the movement amount M are displayed on the x-axis 701 and the diagonal axis 1001. The second movement operation can move the virtual object 511 by one scale at a time. Fig. 9(b) shows the diagonal axis 1001 in Fig. 9(a) as viewed from the x-axis direction. The tilt angle 902 of the diagonal axis 1001 is the same as the tilt angle of the arrow 910 relative to the horizontal direction. Note that the tilt angle 902 is variable, and the change may be continuous or stepwise.

[0053] Although preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and variations are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0054] The disclosure of each embodiment includes the following configurations, methods, systems, and programs. (Configuration 1) An operation means for performing a movement operation to move a virtual object in a virtual reality space or a mixed reality space; a switching means for switching the movement operation of the operation means between a first movement operation that enables continuous movement of the virtual object and a second movement operation that enables stepwise movement of the virtual object; and a direction restriction means for restricting the direction in which the virtual object can be moved by the second movement operation to at least one direction. (Configuration 2) The first movement operation is used to move the virtual object from a source to a destination, 2. The information processing device according to configuration 1, wherein the second movement operation is used to adjust the position of the virtual object at the movement destination. (Configuration 3) The information processing device according to configuration 1 or 2, wherein the maximum distance that can be moved in each step of the second movement operation is shorter than the distance that can be moved in one continuous step of the first movement operation. (Configuration 4) The information processing device according to configuration 3, wherein the maximum distance is determined according to the size of the virtual object. (Configuration 5) A display means capable of displaying the mixed reality space is provided, 5. The information processing device according to configuration 3 or 4, wherein the display means displays a scale indicating the maximum distance. (Configuration 6) The information processing device according to any one of configurations 1 to 5, wherein the switching means switches to the second movement operation when the virtual object moves with the first movement operation and stops for a predetermined time. (Configuration 7) The information processing device described in any one of configurations 1 to 5, characterized in that the switching means switches to the second movement operation when the change in position after the virtual object is moved by the first movement operation remains within a predetermined range for a predetermined time. (Configuration 8) The information processing device according to any one of configurations 1 to 5, wherein the operation means is capable of switching between the first movement operation and the second movement operation with respect to the switching means. (Configuration 9) The direction restriction means restricts the direction in which the virtual object can be moved by the second movement operation to two directions that intersect with each other, and allows the user to select one of the two directions; 9. The information processing device according to any one of configurations 1 to 8, wherein the operation means moves the virtual object in the one direction. (Configuration 10) The information processing device according to configuration 9, wherein the direction restriction means restricts movement of the virtual object in the other of the two directions. (Configuration 11) An information processing device according to configuration 9 or 10, characterized in that the two directions are either the vertical direction and the horizontal direction in the mixed reality space, the horizontal direction and the depth direction in the mixed reality space, or the vertical direction or the horizontal direction in the mixed reality space and an inclined direction inclined with respect to the vertical direction or the horizontal direction. (Configuration 12) A display means capable of displaying the mixed reality space is provided, 12. The information processing device according to any one of configurations 1 to 11, wherein the display means displays the direction restricted by the direction restriction means as a straight line. (Configuration 13) The information processing device according to configuration 12, wherein the display means displays the straight line in an emphasized manner. (Configuration 14) The operation means is a controller configured to be held by a hand of a user who uses the information processing device, The information processing device described in any one of configurations 1 to 13, characterized in that the direction restriction means restricts the direction in which the virtual object can be moved by the second movement operation to at least one direction depending on the orientation or posture of the controller. (Configuration 15) A display means capable of displaying the mixed reality space is provided, 15. The information processing device according to configuration 14, wherein the display means displays a ray according to the direction or attitude of the controller. (Configuration 16) An imaging means capable of capturing an image of the real space; a display means capable of displaying the mixed reality space, the image of the real space includes a hand of a user who uses the information processing device; 14. The information processing device according to any one of configurations 1 to 13, wherein the operation means is a hand of the user displayed in the mixed reality space. (Configuration 17) An imaging means capable of capturing an image of the real space; a display means capable of displaying the mixed reality space, 17. The information processing device according to any one of configurations 1 to 16, which is configured as a head-mounted display. (Method 1) A method for controlling an information processing device, comprising: an operation step of performing a movement operation to move a virtual object within a virtual reality space or a mixed reality space; a switching step of switching the movement operation in the operation step between a first movement operation that enables continuous movement of the virtual object and a second movement operation that enables stepwise movement of the virtual object; a direction restriction step of restricting a direction in which the virtual object can move by the second movement operation to at least one direction. (System 1) An operation step of performing a movement operation to move a virtual object in a virtual reality space or a mixed reality space; a switching step of switching the movement operation in the operation step between a first movement operation that enables continuous movement of the virtual object and a second movement operation that enables stepwise movement of the virtual object; and a direction restriction step of restricting a direction in which the virtual object can move by the second movement operation to at least one direction. (Program 1) A program for causing a computer to execute each means of the information processing device according to any one of configurations 1 to 17. [Explanation of symbols]

[0055] 150 HMD 103 Display 104 CG Control Section 105 Drawing section 107 Control Unit 110 One-Time-Phone-Trend 111 Orientation detection unit 160 Small operating device 301 Virtual Objects

Claims

1. an operating means for performing a movement operation to move a virtual object within a virtual reality space or a mixed reality space; a switching means for switching the movement operation of the operation means between a first movement operation that enables continuous movement of the virtual object and a second movement operation that enables stepwise movement of the virtual object; and a direction restriction means for restricting a direction in which the virtual object can be moved by the second movement operation to at least one direction.

2. the first movement operation is used to move the virtual object from a source to a destination; The information processing apparatus according to claim 1 , wherein the second movement operation is used to adjust the position of the virtual object at the movement destination.

3. 2 . The information processing apparatus according to claim 1 , wherein a maximum distance that can be moved in each step of the second movement operation is shorter than a single continuous distance that can be moved in the first movement operation.

4. The information processing apparatus according to claim 3 , wherein the maximum distance is determined in accordance with the size of the virtual object.

5. a display means capable of displaying the mixed reality space, 4. The information processing apparatus according to claim 3, wherein said display means displays a scale indicating said maximum distance.

6. The information processing apparatus according to claim 1 , wherein the switching means switches to the second movement operation when the virtual object is moved by the first movement operation and then stopped for a predetermined time.

7. 2. The information processing device according to claim 1, wherein the switching means switches to the second movement operation when a change in position after the virtual object is moved by the first movement operation remains within a predetermined range for a predetermined time.

8. 2. The information processing apparatus according to claim 1, wherein the operation means allows the switching means to switch between the first movement operation and the second movement operation.

9. the direction restriction means restricts directions in which the virtual object can be moved by the second movement operation to two directions that intersect with each other, and makes one of the two directions selectable; The information processing apparatus according to claim 1 , wherein the operation means moves the virtual object in the one direction.

10. The information processing apparatus according to claim 9 , wherein the direction restriction means restricts movement of the virtual object in the other of the two directions.

11. 10. The information processing device according to claim 9, wherein the two directions are either a vertical direction and a horizontal direction in the mixed reality space, a horizontal direction and a depth direction in the mixed reality space, or a vertical direction or a horizontal direction in the mixed reality space and an inclined direction inclined relative to the vertical direction or the horizontal direction.

12. a display means capable of displaying the mixed reality space, 2. The information processing apparatus according to claim 1, wherein said display means displays the direction restricted by said direction restriction means as a straight line.

13. 13. The information processing apparatus according to claim 12, wherein the display means displays the straight line in an emphasized manner.

14. the operation means is a controller configured to be held by a hand of a user who uses the information processing device, The information processing apparatus according to claim 1 , wherein the direction restriction means restricts the direction in which the virtual object can be moved by the second movement operation to at least one direction, depending on the orientation or posture of the controller.

15. a display means capable of displaying the mixed reality space, 15. The information processing apparatus according to claim 14, wherein the display means displays a ray according to the direction or attitude of the controller.

16. an imaging means capable of capturing an image of the real space; a display means capable of displaying the mixed reality space, the image of the real space includes a hand of a user who uses the information processing device; 2. The information processing apparatus according to claim 1, wherein the operation means is a hand of the user displayed in the mixed reality space.

17. an imaging means capable of capturing an image of the real space; a display means capable of displaying the mixed reality space, 2. The information processing apparatus according to claim 1, which is configured as a head-mounted display.

18. A method for controlling an information processing device, comprising: an operation step of performing a movement operation to move a virtual object within a virtual reality space or a mixed reality space; a switching step of switching the movement operation in the operation step between a first movement operation that enables continuous movement of the virtual object and a second movement operation that enables stepwise movement of the virtual object; a direction restriction step of restricting a direction in which the virtual object can be moved by the second movement operation to at least one direction.

19. an operation step of performing a movement operation to move a virtual object within a virtual reality space or a mixed reality space; a switching step of switching the movement operation in the operation step between a first movement operation that enables continuous movement of the virtual object and a second movement operation that enables stepwise movement of the virtual object; and a direction restriction step of restricting a direction in which the virtual object can be moved by the second movement operation to at least one direction.

20. 2. A program for causing a computer to execute each means of the information processing apparatus according to claim 1.

Images