Information processing apparatus, information processing method, computer program product, and storage medium

Abstract

The invention provides an information processing apparatus, an information processing method, a computer program product, and a storage medium. The information processing apparatus performs a first setting process for setting a first region within a virtual space based on a line of sight of a user who is viewing the virtual space, and a second setting process for setting a position within the virtual space based on a posture of the user. In the second setting process, after the first region is set in the first setting process, the position to be set is limited within the first region.

Description

Technical Field

[0001] The present invention relates to an information processing device and an information processing method. Background Art

[0002] In a cross-reality (XR) system that enables a user to experience virtual reality and mixed reality with their whole body, a hand controller is used to convert the movement of the user's hand into an action in a virtual space when controlling a display on a head-mounted display (HMD). The HMD is a glasses-type device that the user wears on his / her head and includes a small display.

[0003] In Japanese Patent Application No. 2020-519986, a method is proposed in which an operation start position for an object in a virtual space is selected based on the user's line of sight information, and the object is operated using a hand controller.

[0004] Japanese Patent Application Laid-Open No. 2017-027206 proposes a method of bringing a selected virtual object closer to increase the display size of the virtual object, thereby making operation on the selected virtual object easier.

[0005] However, in the case of the technology disclosed in Japanese Patent Application No. 2020-519986, the position and posture of the hand controller are detected to operate an object, so the user must maintain the position and posture of his / her hand to operate a specific object, which puts a heavy burden on the user.

[0006] In the case of the technology disclosed in Japanese Patent Laid-Open No. 2017-027206, only the selected virtual object is displayed in an enlarged state, so the virtual object on the display screen is changed, and the states of other virtual objects cannot be confirmed while the operation on the selected virtual object is being performed. Summary of the invention

[0007] An object of the present invention is to reduce a burden associated with a user's operation in an information processing device in which the user can perform operations in a virtual space based on the user's gesture information.

[0008] The present invention is an information processing device, comprising: a first setting unit, which is configured to set a first area in the virtual space based on the line of sight of a user who is viewing the virtual space; and a second setting unit, which is configured to set a position in the virtual space based on the posture of the user, wherein, after the first setting unit sets the first area, the second setting unit limits the position to be set within the first area.

[0009] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram for describing an information processing system according to Embodiment 1;

[0011] Figure 2 is a diagram for describing the internal configuration of an HMD or the like according to Embodiment 1;

[0012] Figure 3 is a flowchart of the process according to Example 1;

[0013] Figure 4 is a diagram showing a display example of the HMD according to Embodiment 1;

[0014] Figure 5 is a diagram showing a display example of the HMD according to Embodiment 1;

[0015] Figure 6 is a diagram showing a display example of the HMD according to Embodiment 1;

[0016] Figure 7 is a diagram showing a display example of the HMD according to Embodiment 1;

[0017] Figure 8 is a diagram showing a display example of the HMD according to Embodiment 1;

[0018] Fig. 9 is a diagram showing a display example of an HMD according to Modification 1; and

[0019] Fig.10 : is a diagram showing a display example of an HMD according to Modification 2. DETAILED DESCRIPTION

[0020] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] Example 1

[0022] Reference Figure 1 An information processing system 1 according to Embodiment 1 will be described. The information processing system 1 is an information processing apparatus including an HMD 100 , an image processing device 110 , and a controller 120 .

[0023] The HMD 100 is a head mounted display, which is a head mounted display device (electronic device) that can be mounted on the head of the user. The HMD 100 is a video see-through type HMD, on which a composite image combining a captured image of a range in front of the user captured by the HMD 100 and contents such as computer graphics (CG) according to the posture of the HMD 100 is displayed.

[0024] The image processing device 110 is a control device (image processing equipment; electronic device) for controlling the HMD 100. The image processing device 110 is, for example, a smartphone, a tablet terminal, or a personal computer (PC). The image processing device 110 is connected to the HMD 100 wirelessly or via a cable. The image processing device 110 combines a captured image acquired from the HMD 100 with computer graphics (CG) to generate a composite image of a virtual space, and transmits the composite image to the HMD 100. Each component element of the image processing device 110 may be included in the HMD 100.

[0025] The controller 120 is a device for performing various controls on the HMD 100. When the user operates the controller 120, the image processing device 110 controls the HMD 100 according to the operation performed by the user. The controller 120 is, for example, Figure 1 The ring-type hand controller shown can be worn on the user's finger. If the controller 120 can be worn on the user's finger, the user can freely move their hand while holding the controller 120. The controller 120 also has a button for surrounding an optical tracking pad (hereinafter referred to as OTP) that can detect the moving distance on a plane. The controller 120 wirelessly communicates with the image processing device 110.

[0026] The user causes the HMD 100 to display a menu including a pointer, for example, by long pressing a button. Then, the user touches the OTP with their finger and slides their finger in any direction, thereby placing the pointer on a desired item. Then, the user can confirm the selection of the item by pressing a button. Here, the shape of the controller 120 is annular, but the shape is not limited to annular.

[0027] For example, the controller 120 may be in the shape of a glove-type hand controller that can be worn on their hands. Therefore, the controller 120 may be held by the user's hand or may be worn on the user's hand so that the user can easily operate it.

[0028] Internal configuration of the HMD

[0029] Reference Figure 21 and 2 will now describe the internal configuration of the HMD 100. The HMD 100 includes an HMD control unit 201, an imaging unit 202, a display 203, a posture sensor 204, and a visual line detector 205.

[0030] The HMD control unit 201 controls each component element of the HMD 100. When a composite image (an image generated by combining a captured image when the imaging unit 202 captures a space in front of the user with CG) is acquired from the image processing device 110, the HMD control unit 201 displays the composite image on the display 203 as a display section.

[0031] Therefore, by wearing the HMD 100, the user can view the synthesized image displayed on the display 203. The user can experience various mixed realities as if CG is actually integrated with the real space.

[0032] The imaging unit 202 includes two cameras (imaging device; imaging component). When the user wears the HMD 100, the two cameras are arranged near the positions of the user's left eye and right eye, respectively, to capture the same space as the user is viewing when not wearing the HMD. The image (captured image) generated by capturing the subject (the range in front of the user) by the two cameras is output to the image processing device 110. The two cameras of the imaging unit 202 can also obtain information related to the distance from the two cameras to the subject as distance information based on the distance measurement of the stereo camera.

[0033] The display 203 displays the synthesized image. The display 203 includes, for example, a liquid crystal display panel or an organic EL panel. In a state where the user wears the HMD 100, images corresponding to the left eye and the right eye of the user are displayed respectively.

[0034] A device using a semi-transmissive half-mirror can also be used for the display 203. For example, the display 203 can display an image as if the CG is directly superimposed on the real space seen through the half-mirror. In this case, the HMD 100 is called an optical see-through HMD (not a video see-through HMD). In addition, the display 203 can display only an image of a completely virtual space without using a captured image of the real space (or the real space itself). In this embodiment, the "virtual space" refers not only to a completely virtual space such as virtual reality (VR), but also to a virtual space mixed with a real space such as mixed reality (MR).

[0035] The posture sensor 204 detects the posture and position of the HMD 100. The posture sensor 204 includes, for example, an inertial measurement unit (IMU). The posture sensor 204 outputs information on the posture of the HMD 100 to the image processing device 110 as posture information.

[0036] The sight line detector 205 is a sight line detector for detecting sight line information of the user wearing the HMD 100. The sight line detector 205 includes, for example, an imaging unit (e.g., a camera) installed in the HMD 100. The sight line detector 205 uses the imaging component to capture the eyes of the user and analyzes the captured image to detect the sight line.

[0037] Internal configuration of the image processing device

[0038] Reference Figure 2 The internal configuration of the image processing apparatus 110 will be described. The image processing apparatus 110 includes a control unit 211, a content DB 212, and a communication unit 213.

[0039] The control unit 211 receives an image (captured image) acquired by the camera unit 202 of the HMD 100, posture information acquired by the posture sensor 204, and line of sight information acquired by the line of sight detector 205 from the HMD 100. Then, the control unit 211 combines the captured image and arbitrary CG, and thereby generates a composite image to be displayed on the HMD 100. The control unit 211 can also interpret instructions from the user based on the line of sight information.

[0040] The control unit 211 sends the synthesized image to the HMD control unit 201 of the HMD 100. In addition, based on the information (distance information and posture information) acquired by the HMD 100, the control unit 211 controls the position, direction, and size of CG in the synthesized image. For example, in a case where a virtual object indicated by CG is arranged near a specific object existing in the real space in the virtual space represented by the synthesized image, the control unit 211 increases the size of the virtual object (CG) as the distance between the specific object and the camera unit 202 becomes shorter. By controlling the position, direction, and size of CG like this, the control unit 211 can generate a synthesized image as if an object of CG that is not arranged in the real space is actually arranged in the real space.

[0041] The content DB 212 is a storage unit that stores information related to CG, etc. The control unit 211 can select a CG read from the content DB 212 (ie, a CG used to generate a synthesized image).

[0042] Internal configuration of the controller

[0043] Reference Figure 21 and 12. The internal configuration of the controller 120 is described below. The controller 120 includes a controller control unit 221, an operation unit 222, a communication unit 223, and a motion detector 224.

[0044] The controller control unit 221 controls each component of the controller 120 .

[0045] The operation unit 222 includes buttons for surrounding the OTP. Information (operation information) related to operations such as pressing the OTP and sliding a finger at the operation unit 222 is transmitted to the image processing apparatus 110 via the communication unit 223.

[0046] For example, the user can move the pointer displayed on the HMD 100 to a predetermined position by sliding a finger on the OTP. In addition, the user can instruct the HMD 100 and the image processing device 110 to perform a specific process by pressing a button at the OTP. Therefore, the user can control the HMD 100 by a combination of sliding a finger on the OTP and pressing a button at the OTP. Specifically, the user causes the HMD 100 to display a menu by long pressing the button at the OTP, places the pointer at a desired position and presses the button at the OTP, thereby selecting the position. The operation unit 222 may include any operation member instead of the OTP as long as the user can operate by physical contact with the operation member. For example, the operation unit 222 may include any one of a touch pad, a touch panel, a cross key, a joystick, and a track pad device instead of the OTP.

[0047] The communication unit 223 performs wireless communication with the image processing device 110 (communication unit 213 ).

[0048] The motion detector 224 includes an inertial measurement unit (IMU: inertial sensor) that can detect inertial information (spatial movement distance and angle) of the controller 120. The motion detector 224 may include any device as long as the device does not hinder the miniaturization of the controller 120 and can detect inertial information (information such as displacement, velocity, and acceleration).

[0049] As described above, the camera unit 202 of the HMD 100 is a sight line detection unit for detecting sight line information of the user. The motion detector 224 of the controller 120 is a gesture detection unit for detecting gesture information of the user. The image processing device 110 is an image processing unit for generating an image of a virtual space based on sight line information and gesture information.

[0050] Reference Figure 3 Flowchart and Figures 4 to 7 The processing according to Embodiment 1 will be described.

[0051] When the operation starts, Figure 4 As shown, the control unit 211 causes the display 203 to display a plurality of virtual objects 402, 403, and 404 in the image display area 401. The control unit 211 generates the virtual objects 402 to 404 based on the information stored in the content DB 212. When the user selects any one of the displayed virtual objects, the control unit 211 generates a state (selection state) in which a more detailed item can be selected for the virtual object. Here, the control unit 211 can combine the image captured by the camera unit 202 with the virtual object and display the composite image.

[0052] In step S1001, the control unit 211 detects the line of sight of the user wearing the HMD 100 based on the line of sight information detected by the line of sight detector 205. The control unit 211 calculates the intersection of the line of sight and the virtual object arranged in the virtual space, sets (determines) the line of sight position of the user in the virtual space, and displays a line of sight cursor on the virtual object. The line of sight cursor is an image used to indicate the position of the user's operation target. Figure 5 4 is an example in which the line of sight cursor 410 is displayed on the virtual object 403. The line of sight cursor 410 has an outer frame and a center point, and in the stage of detecting the line of sight, the control unit 211 displays the outer frame of the line of sight cursor 410 by a solid line. The configuration of the image of the line of sight cursor is not limited to this example.

[0053] In step S1002, the control unit 211 determines whether there is a position in the virtual space that the user is gazing at. For example, in a case where the state of the user's line of sight position in the virtual space set based on the line of sight information detected by the line of sight detector 205 remains in a predetermined range in the virtual space for a predetermined time, the control unit 211 determines that there is a position in the range that the user is gazing at. The control unit 211 sets the position that the user is gazing at in the virtual space as the gaze position. In the case where there is a virtual object at the gaze position, the control unit 211 determines that the user is gazing at the virtual object. If it is determined that there is a gaze position in the virtual space, the control unit 211 performs the processing in step S1003, otherwise, the control unit 211 returns to the processing in step S1001 and repeats the line of sight detection processing.

[0054] In step S1003, the control unit 211 sets the operable range (first area) in the virtual space based on the gaze position of the user. The control unit 211 detects a virtual object existing at the gaze position of the user, and sets the area where the virtual object exists as the operable range based on the shape information of the virtual object. Therefore, the image processing device 110 includes a first setting component for setting the first area in the virtual space based on the line of sight of the user who is viewing the virtual space, and a selection component for selecting an object arranged in the virtual space based on the line of sight of the user. Figure 5 In the embodiment, for example, the control unit 211 sets the area where the virtual object 403 exists at the gaze position indicated by the cursor 410 as the operable range. Here, the entire area where the virtual object 403 exists may be set as the operable range, or only the area around the user's gaze position in the area where the virtual object 403 exists may be set as the operable range.

[0055] In step S1004, the control unit 211 displays a line indicating a virtual object. The line is an image indicating the position of the user's operation target in the virtual space, and is displayed as an image of light, for example. The user can use the posture of their hand or finger to specify the position in the virtual space. Based on the posture information detected by the controller 120 as a posture detection component worn on the user's hand 405, the control unit 211 sets the specified position in the virtual space specified by the user using the posture. Therefore, the image processing device 110 includes a second setting component for setting the position in the virtual space based on the user's posture. In Example 1, the specified position is set based on the posture of the controller 120 as a hand controller. Then, the control unit 211 sets the position of the front end of the line and the position and length direction of the line based on the specified position. Figure 6 Display example of an indication line Here, a line 420 is displayed from the controller 120 worn on the user's hand 405 to the area where the virtual object 403 exists.

[0056] The control unit 211 sets the initial position of the front end of the line to a predetermined position in the operable range. In Embodiment 1, the operable range is set in the area where the virtual object being watched is determined to exist, so the control unit 211 sets the initial position of the front end of the line to a predetermined position such as the center or upper end of the virtual object being watched. The initial position of the front end of the line is not limited thereto, and may be, for example, a position predetermined for each virtual object.

[0057] Furthermore, the control unit 211 limits the movement of the leading end position of the wire to within the operable range set in step S1003. Figure 6In the case of the example in , the control unit 211 restricts the movement of the front end position of the line so that it can move only within the area where the virtual object 403 determined to be being watched exists. Therefore, the image processing device 110 includes a second setting component, and after setting the first area (operable range) in the virtual space based on the user's line of sight, the second setting component restricts the position to be set in the virtual space based on the user's gesture within the first area. Specifically, it is assumed that: after the operable range is set in step S1003, the user's designated position in the virtual space set based on the gesture information detected by the controller 120 is moved out of the operable range. In this case, the control unit 211 sets the restricted position (the restricted position is a position restricted within the operable range), and generates an image so that an image (the front end of the line) indicating the position of the user's operation target is displayed at the restricted position. On the other hand, in the case where the user's designated position in the virtual space set based on the gesture information detected by the controller 120 is within the operable range, the control unit 211 generates an image (the front end of the line) indicating the position of the user's operation target so that the image is displayed at the designated position.

[0058] In addition, the control unit 211 generates an image of a virtual space so that an image indicating an operable range is displayed. Figure 6 In the example of FIG. 4 , the operable range 430 is displayed as a dotted line frame image surrounding the virtual object 403 determined to be being looked at. By displaying an image indicating the operable range, the user can recognize that the position to which the front end of the line 420 can be moved using the controller 120 is limited within the operable range.

[0059] In addition, after setting the operable range in step S1003, the control unit 211 generates an image of the virtual space so that the information indicating the line of sight position (line of sight cursor) set based on the line of sight information detected by the line of sight detector 205 is not displayed. Figure 6 In the case of the example, Figure 5 The gaze cursor 410 displayed in FIG. 1 is hidden.

[0060] In step S1005 , the control unit 211 detects the user's line of sight information using the line of sight detector 205 .

[0061] In step S1006, the control unit 211 determines whether the user's line of sight position in the virtual space set based on the user's line of sight information detected in step S1005 is outside the operable range. If it is determined that the user's line of sight position is outside the operable range (step S1006: Yes), the control unit 211 performs the processing in step S1007. If it is determined that the user's line of sight position is within the operable range (step S1006: No), the control unit 211 performs the processing in step S1008.

[0062] In step S1007 , the control unit 211 displays again the line-of-sight cursor that was hidden in step S1004 . Figure 7 Indicates a display example in the case where the line of sight position moves out of the operable range after the operable range is set, and moves onto the virtual object 402. Figure 7 As shown, in Figure 6 The hidden sight cursor is in Figure 7 The control unit 211 may make the sight cursor 411 displayed when the user's sight position moves out of the operable range after the operable range is set different from the sight cursor displayed when the operable range is not set. Figure 7 In the example of FIG. 1 , the sight cursor 411 is an image composed of a dotted outer frame and a center point, which is different from the image composed of a solid outer frame and a center point. Figure 5 By displaying the line of sight cursor in a different form like this, it is possible to notify the user when the line of sight position moves out of the operable range after the operable range is set.

[0063] In step S1008 , the control unit 211 maintains the state in which the line-of-sight cursor is hidden in step S1004 .

[0064] In step S1009, the control unit 211 sets the display of an image (line) indicating the position of the user's operation target in the virtual space based on the operable range and the position information and posture information of the controller 120. The control unit 211 controls the display of the image (line) indicating the position of the user's operation target in the virtual space according to whether the designated position in the virtual space determined by the user's posture, which is set based on the posture information of the controller 120, is outside the operable range. Specifically, when the designated position determined by the user's posture is within the operable range, the control unit 211 displays the front end of the line at the designated position, and when the designated position is outside the operable range, the control unit 211 displays the front end of the line at a restricted position restricted within the operable range. The restricted position is, for example, the position closest to the designated position on the boundary line of the operable range. Thus, even if the user uses the controller 120 to perform an operation that causes the front end of the line to deviate from the operable range, the front end position of the line in the display will not move out of the boundary of the operable range. The state in which the front end of the line is displayed at the restricted position when the designated position moves out of the operable range is referred to as a "restricted state".

[0065] In step S1010, the control unit 211 determines whether the front end position of the wire is in a restricted state. If it is determined that the front end position is in a restricted state, the control unit 211 performs the processing in step S1011, otherwise, performs the processing in step S1012.

[0066] In step S1011, the control unit 211 switches the display of the operable range to a warning display. Figure 7 In the example of FIG. 4 , the image used to indicate the operable range 431 is an image of a dotted frame. Figure 6 The image for indicating the operable range 430 when not in the restricted state is different from the image for indicating the operable range 430 shown. The front end position of the line 421 is displayed within the operable range 431, but the starting point of the line 421 is displayed at a position away from the controller 120. The control unit 211 also makes the image for indicating the line in the restricted state different from the image for indicating the line not in the restricted state. Figure 7 In the example case, the line 421 in the restricted state is shown as Figure 6 The line 420 in the normal state (not in the restricted state) is shown as a different dotted line. After the operable range is set like this, by making the image and the image of the line indicating the operable range different between the case where the designated position is out of the operable range and the case where the designated position is within the operable range, the user can be notified when the designated position is out of the operable range.

[0067] The display method of the line in the restricted state is not limited to this. Figure 8As shown, the line 422 may be displayed as a curve, with the position of the starting point of the line being set to the controller 120 .

[0068] In step S1012, the control unit 211 switches the display of the operable range to the normal display. Figure 6 The operable range 430 is shown in FIG.

[0069] In step S1013, the control unit 211 determines whether the state in which the user's line of sight position is outside the operable range (condition A) and the user's designated position is outside the operable range (condition B) continues for a specified time. Condition B can be interpreted as a condition in which the front end position of the line is in a restricted state. The determination method for condition A is the same as the determination method in step S1006. The determination method for condition B is the same as the determination method in step S1010. If it is determined that conditions A and B have continued for a predetermined time, the control unit 211 performs the processing of step S1014, or otherwise, returns to step S1005 and repeats the processing.

[0070] In step S1014, the control unit 211 cancels the operable range set in step S1003, and generates an image that displays an image (front end of a line) indicating the position of the user's operation target in the virtual space at a designated position based on the user's gesture. Figure 6 The operable range 430 and line 420 shown in FIG. 4 are hidden, and the Figure 5 The sight cursor 410 shown in FIG. Figure 5 The image shown in .

[0071] According to Embodiment 1, after the virtual object 403 is selected, the operation range using the controller 120 can be limited to the operable range set in the area where the virtual object 403 exists. Therefore, the user does not need to perform an operation to maintain the front end position of the line within the range where the virtual object to be operated exists using the controller 120. As a result, the burden on the user can be reduced.

[0072] In the example described in Embodiment 1, a virtual object present at the user's gaze position is selected, and the operable range is set based on the selected virtual object, but the method for setting the operable range is not limited to this. For example, a predetermined range centered on the user's gaze position may be set as the operable range. In addition, the method for setting the operable range based on the virtual object is not limited to the method for setting the range in which the virtual object exists as the operable range as described above. For example, a range including a predetermined range around the area in which the virtual object exists may be set as the operable range, or a portion of the area in which the virtual object exists may be set as the operable range.

[0073] Modification 1

[0074] In Modification 1, the following description will be given in which the Figure 3 An example of the case where the condition for determining gaze in step S1002 of . In Embodiment 1, when the line of sight position of the user in the virtual space set based on the line of sight information detected by the line of sight detector 205 exists in a predetermined range for a predetermined time, it is determined that the user is gazing at the gaze position within the range. In addition, when the line of sight position of the user is in a range where a specific virtual object exists for a predetermined time, it is determined that the user is gazing at the virtual object. In Modification 1, when the designated position determined by the user set based on the posture information detected by the controller 120 exists in a predetermined range for a predetermined time, it is also determined that the user is gazing at the gaze position within the range. In other words, if the state of the line of sight position and the designated position within the predetermined range in the virtual space continues for a predetermined time, the control unit 211 of the image processing device 110 sets the gaze position within the range. In addition, if the state of the line of sight position and the designated position of the user within the range where a specific virtual object exists continues for a predetermined time, the control unit 211 determines that the user is gazing at the virtual object.

[0075] exist Fig. 9 In addition to Figure 5 In addition to the sight cursor 410 shown in , a line 420 from the controller 120 toward the area in the virtual object 403 is also displayed. In the case where the state where the front end of the sight cursor 410 and the line 420 exist on the virtual object 403 continues for a predetermined time, the control unit 211 determines that the virtual object 403 is being stared at, and performs the processing in step S1003. Here, the predetermined time (determination threshold) can be set shorter than that in the case of Embodiment 1. In the case of performing this staring determination, the control unit 211 displays the sight cursor 410 and the line 420 in the image display area 401 at the time of step S1001.

[0076] The gaze may be determined based only on the posture information detected by the controller 120 without using the line of sight information detected by the line of sight detector 205. In this case, the control unit 211 sets a designated position using the user's posture in the virtual space based on the posture information of the controller 120, and selects an object arranged in the virtual space based on the designated position. Then, the control unit 211 sets the operable range in the virtual space based on the information related to the selected object. In this case, the image processing device 110 includes a second setting component for setting a position in the virtual space based on the posture of the user who is viewing the virtual space and a selection component for selecting an object arranged in the virtual space based on the set position. The image processing device 110 also includes a third setting component for setting an area in the virtual space based on information related to the object. Just like in Embodiment 1, the control unit 211 may set the operable range based on the shape information of the object. In addition, after setting the operable range, the control unit 211 may limit the designated position determined by the user's posture within the operable range. In addition, after setting the operable range, the control unit 211 may control so that the information (line of sight cursor) indicating the line of sight position of the user in the virtual space is no longer displayed. When the operable range is set and the user's line of sight moves out of the operable range thereafter, the control unit 211 can control so that the information (line of sight cursor) indicating the user's line of sight in the virtual space is displayed again. In addition, when the operable range is set and the user's line of sight deviates from the operable range thereafter, the control unit 211 can, for example, notify the user of the state by changing the drawing form of the line of sight cursor. In addition, when the operable range is set and the position in the virtual space specified by the user's gesture moves out of the operable range thereafter, the control unit 211 can, for example, notify the user by changing the image and line image indicating the operable range. In addition, when the operable range is set and the designated position determined by the user's gesture thereafter is outside the operable range, if the state of the user's line of sight outside the operable range thereafter continues for a predetermined time, the control unit 211 can cancel the setting of the operable range. When the designated position determined by the user's gesture set based on the gesture information detected by the controller 120 exists in the predetermined range for a predetermined time, the control unit 211 can determine that the user is looking at the gaze position within the range. Furthermore, when the designated position determined by the user's gesture maintains a specific state of being located within the range where a specific virtual object exists for a predetermined time, the control unit 211 may determine that the user is looking at the virtual object.Furthermore, when the designated position determined by the user's gesture is within the range where a specific virtual object exists and a button included in the controller 120 is operated in this state, the control unit 211 may determine that the user is looking at the virtual object.

[0077] Modification 2

[0078] In the examples described in Embodiment 1 and Modification 1, display of lines is controlled based on position information and posture information of controller 120. In Modification 2, a method for detecting a user's hand from a captured image and controlling a line based on detected information of the user's hand will be described.

[0079] exist Fig.10 , a state in which a line 420 in the area from the fingertip of the user's hand 406 to the virtual object 403 is displayed is shown. The control unit 211 detects the user's hand from the image captured by the camera unit 202, and acquires the position information and posture information of the user's hand. Then, the control unit 211 sets the display position and direction of the line based on the position information and posture information of the user's hand, and displays the line 420 in the image display area 401.

[0080] The control of the line 420 based on the position information and posture information of the controller 120 described in Embodiment 1 and Modification 1 may be replaced by the control of the line 420 based on the position information and posture information of the user's hand 406 .

[0081] Although the preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments, and may be modified and altered in various ways within the scope of the spirit of the present invention.

[0082] According to the present invention, in an information processing device in which a user can perform operations in a virtual space based on the user's gesture information, the burden associated with the user's operations can be reduced.

[0083] Note that the various types of control described above may be processes performed by one hardware (e.g., a processor or a circuit) or in other ways. Processing may be shared between multiple hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) to perform control of the entire device.

[0084] In addition, the above-mentioned processor is a processor in a broad sense, and includes a general-purpose processor and a special-purpose processor. Examples of general-purpose processors include a central processing unit (CPU), a microprocessing unit (MPU), a digital signal processor (DSP), etc. Examples of special-purpose processors include a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), and a programmable logic device (PLD). Examples of PLDs include a field programmable gate array (FPGA) and a complex programmable logic device (CPLD), etc.

[0085] The above-mentioned embodiments (including modified examples) are only examples. Any configuration obtained by appropriately modifying or changing some configurations of the embodiments within the scope of the subject matter of the present invention is also included in the present invention. The present invention also includes other configurations obtained by appropriately combining various features of the embodiments.

[0086] Other embodiments

[0087] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0088] Although the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: a first setting unit configured to set a first area in the virtual space based on a line of sight of a user who is viewing the virtual space; as well as a second setting unit configured to set a position in the virtual space based on the user's gesture, After the first setting unit sets the first area, the second setting unit limits the position to be set within the first area.

2. The information processing device according to claim 1, wherein: The first setting unit sets the first area in the virtual space when the user gazes at the predetermined area for a predetermined time.

3. The information processing device according to claim 1, further comprising a selection unit configured to select an object arranged in the virtual space based on a line of sight of the user, in, The first setting unit sets the first area based on information related to the object.

4. The information processing device according to claim 3, wherein: The first setting unit sets the first area based on information about a shape of the object.

5. The information processing device according to any one of claims 1 to 4, wherein: After the first setting unit sets the first area, information indicating the position of the user's line of sight is not displayed in the virtual space.

6. The information processing device according to claim 5, wherein: After the first setting unit sets the first area, when the line of sight of the user moves out of the first area, information indicating the position of the line of sight of the user is displayed again in the virtual space.

7. The information processing device according to any one of claims 1 to 4, wherein: After the first setting unit sets the first area, when the user's sight moves out of the first area, such a state is notified to the user.

8. The information processing device according to any one of claims 1 to 4, wherein: After the first setting unit sets the first area, in a case where the position in the virtual space specified based on the gesture of the user moves out of the first area, such a state is notified to the user.

9. The information processing device according to any one of claims 1 to 4, wherein: After the first setting unit sets the first area, if the position in the virtual space specified based on the user's gesture is outside the first area and the user's line of sight is outside the first area for a predetermined time, the first setting unit cancels the setting of the first area.

10. The information processing device according to any one of claims 1 to 4, wherein: In a case where the first setting unit sets the first area, the second setting unit maintains the position on the boundary of the first area even if the user has a gesture based on which the position is to be set outside the first area. 11 . The information processing device according to claim 1 , further comprising an acquisition unit configured to acquire a posture of the user based on a posture of a controller worn by the user or held by the user. 12 . The information processing apparatus according to claim 1 , further comprising an acquisition unit configured to acquire the user's gesture from a captured image capturing a real space.

13. A method for controlling an information processing device, comprising: A first setting step for setting a first area in the virtual space based on a line of sight of a user who is viewing the virtual space; as well as A second setting step is used to set the position in the virtual space based on the user's posture, Wherein, in the second setting step, after the first area is set in the first setting step, the position to be set is limited within the first area. 14 . A computer program product comprising a control program for causing a computer to execute each step of the control method of an information processing device according to claim 13 . 15 . A computer-readable storage medium storing a control program for causing a computer to execute each step of the control method of the information processing device according to claim 13 .

Images