Information processing device, information processing method, and program

Abstract

To reduce a burden related to user's operation in an information processing device for enabling user's operation within a virtual space on the basis of user's attitude information.SOLUTION: An information processing device includes first setting means for setting an area within a virtual space on the basis of the line of sight of a user looking at the virtual space, and second setting means for setting a position within the virtual space on the basis of the user's attitude. The second setting means limits a position to be set within an area after the first setting means sets the area.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In cross reality (XR) systems that allow users to experience virtual reality and mixed reality, hand controllers are used to convert the movements of the user's hands into actions in the virtual space when controlling the display of a head-mounted display (HMD). The HMD is a glasses-type device equipped with a small display that the user wears on his or her head.

[0003] Patent Document 1 proposes a method of selecting an operation start position of an object in a virtual space from a user's line of sight information and operating the object using a hand controller. Patent Document 2 also proposes a method of making a selected virtual object easier to operate by drawing it closer and increasing its display size. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2020-519986 [Patent Document 2] JP 2017-027206 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology disclosed in Patent Document 1, the position and posture of the hand controller are detected to operate an object, so the user must maintain the position and posture of the hand in order to operate a specific object, which places a heavy burden on the user. Also, in the technology disclosed in Patent Document 2, only the selected virtual object is enlarged and displayed, so the arrangement of the virtual object within the display screen is changed, and the state of other virtual objects cannot be confirmed during an operation on the selected virtual object.

[0006] The present invention aims to reduce the burden on a user relating to operation in an information processing device that allows the user to operate the device in a virtual space based on posture information of the user. [Means for solving the problem]

[0007] The present invention includes a first setting means for setting an area in a virtual space based on a line of sight of a user viewing the virtual space; A second setting means for setting a position in the virtual space based on a posture of the user; having The information processing device is characterized in that, after the area is set by the first setting means, the second setting means limits the setting position to within the area.

[0008] The present invention also provides a virtual space control system comprising: a second setting means for setting a position in the virtual space based on a posture of a user viewing the virtual space; a selection means for selecting an object arranged in the virtual space based on the position set by the second setting means; a third setting means for setting an area in the virtual space based on information about the object; having After the area is set by the third setting means, the second setting means The information processing device is characterized in that the position where the image is displayed is limited to within the area. Effect of the Invention

[0009] According to the present invention, in an information processing device that allows a user to perform operations in a virtual space based on posture information of the user, the burden associated with the operation of the user can be reduced. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 2] 2 is a diagram for explaining the internal configuration of an HMD according to the first embodiment. FIG. [Diagram 3] 2 is a flowchart of a process according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing one display example of the HMD according to the first embodiment. [Figure 9] FIG. 13 is a diagram showing one display example of an HMD according to Modification 1. [Figure 10] FIG. 13 is a diagram showing one display example of an HMD according to Modification 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0012] Example 1 An information processing system 1 according to a first embodiment will be described with reference to Fig. 1. The information processing system 1 is an information processing device including an HMD 100, an image processing device 110, and a controller 120.

[0013] The HMD 100 is a head-mounted display that is a head-mounted display device (electronic device) that can be worn on the head of a user. The HMD 100 is a video see-through type HMD, and displays a composite image that combines an image captured by the HMD 100 of the area in front of the user with content such as CG (computer graphics) in a form that corresponds to the posture of the HMD 100.

[0014] The image processing device 110 is a control device (information processing device; electronic device) that controls the HMD 100. The image processing device 110 is, for example, a smartphone, a tablet terminal, or a PC (personal computer). The image processing device 110 is connected to the HMD 100 wirelessly or by wire. The image processing device 110 generates a composite image of a virtual space by combining a captured image acquired from the HMD 100 with a CG (graphics image), and transmits the composite image to the HMD 100. Note that each component of the image processing device 110 may be included in the HMD 100.

[0015] The controller 120 is a device for performing various controls of the HMD 100. When a user operates the controller 120, the image processing device 110 controls the HMD 100 in response to the user's operation. The controller 120 is, for example, a hand controller having a finger ring shape that can be worn on a user's finger, as shown in FIG. 1. If the controller 120 can be worn on a user's finger, the user can freely move his / her hand while holding the controller 120. The controller 120 also has a button with a built-in optical track pad (hereinafter referred to as "OTP") that can sense the amount of planar movement. The controller 120 communicates with the image processing device 110 wirelessly.

[0016] For example, the user can display a menu including a pointer on the HMD 100 by pressing and holding a button. The user then places a finger on the OTP and slides it in any direction to move the pointer to a desired item. The user then presses a button to perform a confirmation action to confirm the selection of the item. Note that, although the controller 120 has been described as having a ring-like shape, the shape is not limited to this.

[0017] For example, the shape of the controller 120 may be a hand controller that is shaped like a glove and can be worn on the hand. In this way, the controller 120 may be in a form that can be held by the user's hand or worn on the hand, so that the user can easily use it.

[0018] (About the internal structure of the HMD) The internal configuration of the HMD 100 will be described with reference to Fig. 2. The HMD 100 includes an HMD control unit 201, an imaging unit 202, an image display unit 203, a posture sensor unit 204, and a gaze detection unit 205.

[0019] The HMD control unit 201 controls each component of the HMD 100. When the HMD control unit 201 acquires a composite image (an image obtained by combining a captured image of the space in front of the user captured by the imaging unit 202 and CG) from the image processing device 110, the HMD control unit 201 displays the composite image on the image display unit 203, which is a display means.

[0020] Therefore, by wearing the HMD 100, the user can view the composite image displayed on the image display unit 203. The user can experience various mixed realities, such as CG blended into real space.

[0021] The imaging unit 202 includes two cameras (imaging devices, imaging means). The two cameras are disposed near the positions of the left and right eyes of the user when wearing the HMD 100 in order to capture an image of a space similar to the space the user sees without wearing the HMD. Images (captured images) captured by the two cameras of a subject (the range in front of the user) are output to the image processing device 110. In addition, the two cameras in the imaging unit 202 can acquire information on the distance from the two cameras to the subject as distance information by measuring distances with a stereo camera.

[0022] The image display unit 203 displays the composite image. The image display unit 203 has, for example, a liquid crystal panel or an organic EL panel, etc. When the user is wearing the HMD 100, images corresponding to the left and right eyes of the user are displayed, respectively.

[0023] It is also possible to use a device using a semi-transmissive half mirror for the image display unit 203. For example, the image display unit 203 may display an image so that CG is directly superimposed on the real space seen through the half mirror. In this case, the HMD 100 is called an optical see-through type HMD (rather than a video see-through type). The image display unit 203 may display only an image of a complete virtual space without using a captured image of the real space (or the real space). It is to be noted that in this specification, not only a complete virtual space such as virtual reality (VR) but also a virtual space that is mixed with a real space such as mixed reality (MR) is called a virtual space.

[0024] The orientation sensor unit 204 detects the orientation and position of the HMD 100. The orientation sensor unit 204 includes, for example, an inertial measurement unit (IMU). The orientation sensor unit 204 outputs information on the orientation of the HMD 100 to the image processing device 110 as orientation information.

[0025] The gaze detection unit 205 is a gaze detection unit that detects gaze information of a user wearing the HMD 100. The gaze detection unit 205 has, for example, an imaging unit such as a camera provided in the HMD 100. The gaze detection unit 205 captures an image of the user's eyes with the imaging unit and analyzes the captured image. This allows for gaze detection.

[0026] (Internal configuration of image processing device) The internal configuration of the image processing device 110 will be described with reference to Fig. 2. The image processing device 110 includes a control unit 211, a content DB 212, and a communication unit 213.

[0027] The control unit 211 receives from the HMD 100 an image (captured image) acquired by the imaging unit 202 of the HMD 100, posture information acquired by the posture sensor unit 204, and gaze information acquired by the gaze detection unit 205. Then, the control unit 211 synthesizes the captured image with any CG to generate a composite image to be displayed on the HMD 100. The control unit 211 can also interpret instructions from the user based on the gaze information.

[0028] The control unit 211 transmits the composite image to the HMD control unit 201 of the HMD 100. The control unit 211 controls the position, orientation, and size of the CG in the composite image based on information (distance information and orientation information) acquired by the HMD 100. For example, when placing a virtual object represented by the CG near a specific object existing in real space in the virtual space represented by the composite image, the control unit 211 makes the virtual object (CG) larger as the distance between the specific object and the imaging unit 202 becomes closer. By controlling the position, orientation, and size of the CG in this manner, the control unit 211 can generate a composite image in which a CG object not placed in real space appears as if it were placed in real space.

[0029] The content DB 212 is a storage unit that stores information on CG, etc. The control unit 211 can switch the CG read from the content DB 212 (that is, the CG used to generate a composite image).

[0030] (Internal structure of the controller) The internal configuration of the controller 120 will be described with reference to Fig. 2. The controller 120 has a controller control section 221, an operation section 222, a communication section 223, and a motion detection section 224.

[0031] The controller control unit 221 controls each component of the controller 120 .

[0032] The operation unit 222 includes a button with an OTP built in. Information (operation information) regarding pressing or sliding a finger on the OTP on the operation unit 222 is transmitted to the image processing device 110 via the communication unit 223.

[0033] For example, the user can move the pointer displayed on the HMD 100 to a predetermined position by sliding the finger on the OTP. Also, the user can instruct the HMD 100 and the image processing device 110 to perform a specific process by pressing a button on the OTP. In this way, the user can control the HMD 100 by a combination of sliding the finger on the OTP and pressing a button. More specifically, the user can control the HMD 100 to display a menu by pressing and holding the OTP button, for example, and select the desired location by aligning the pointer with the desired location and then pressing the OTP button. The operation unit 222 may include any operation member instead of the OTP as long as the user can perform the operation by physical contact. For example, the operation unit 222 may include any of a touch pad, a touch panel, a cross key, a joystick, and a track pad device instead of the OTP.

[0034] The communication unit 223 wirelessly communicates with the image processing device 110 (the communication unit 213).

[0035] The motion detection unit 224 detects inertial information (amount of spatial movement and angle) of the controller 120. The motion detection unit 224 may include any device that does not impede the miniaturization of the controller 120 and can detect inertial information (information such as positional displacement, speed, or acceleration).

[0036] As described above, the imaging unit 202 of the HMD 100 is a gaze detection unit that detects gaze information of the user. The motion detection unit 224 of the controller 120 is a posture detection unit that detects posture information of the user. The image processing device 110 is an image processing unit that generates an image of the virtual space based on the gaze information and posture information.

[0037] The process in the first embodiment will be described with reference to the flowchart in FIG. 3 and FIGS.

[0038] When the operation starts, the control unit 211 causes the image display unit 203 to display a plurality of virtual objects 402, 403, and 404 in the image display area 401, as shown in Fig. 4. The control unit 211 generates the virtual objects 402 to 404 based on information stored in the content DB 212. When the user selects one of the displayed virtual objects, the control unit 211 sets the virtual object in a state in which more detailed items can be selected (selected state). At this time, the control unit 211 may compositely display the image captured by the imaging unit 202 and the virtual object.

[0039] In step S1001, the control unit 211 detects the gaze of the user wearing the HMD 100 based on the gaze information detected by the gaze detection unit 205. The control unit 211 calculates an intersection between the detected gaze information and a virtual object in the virtual space, sets (determines) the gaze position of the user in the virtual space, and displays a gaze cursor on the virtual object. The gaze cursor is an image indicating the position that the user is operating. FIG. 5 shows an example in which the gaze cursor 410 is displayed on the virtual object 403. The gaze cursor 410 has an outer frame and a center point, and at the stage where the gaze is detected, the control unit 211 displays the outer frame of the gaze cursor 410 in a solid line. The configuration of the gaze cursor image is not limited to this example.

[0040] In step S1002, the control unit 211 determines whether there is a position where the user is gazing in the virtual space. For example, when the state where the gaze position of the user in the virtual space set based on the gaze information detected by the gaze detection unit 205 is within a certain range in the virtual space continues for a predetermined time, the control unit 211 determines that the position where the user is gazing is within the range. The control unit 211 sets the position where the user is gazing in the virtual space as the gaze position. When a virtual object exists at the gaze position, the control unit 211 determines that the user is gazing at the virtual object. When it is determined that there is a gaze position in the virtual space, the control unit 211 executes the process of step S1003, and when it is not, the control unit 211 returns to the process of step S1001 and repeats the gaze detection process.

[0041] In step S1003, the control unit 211 sets an operable range in the virtual space based on the user's gaze position. The control unit 211 detects a virtual object present at the user's gaze position, and sets the area in which the virtual object exists as the operable range based on shape information of the virtual object. Therefore, the image processing device 110 has a first setting means for setting an area in the virtual space based on the line of sight of the user looking at the virtual space, and a selection means for selecting an object arranged in the virtual space based on the line of sight of the user. Taking FIG. 5 as an example, the control unit 211 sets the area in which the virtual object 403 exists at the gaze position indicated by the gaze cursor 410 as the operable range. At this time, the entire area in which the virtual object 403 exists may be set as the operable range, or only the periphery of the user's gaze position in the area in which the virtual object 403 exists may be set as the operable range.

[0042] In step S1004, the control unit 211 displays a ray for operating a virtual object. A ray is an image showing a position in the virtual space that the user is operating, and is displayed as an image like a ray, for example. The user can specify a position in the virtual space by the posture of the hand or fingers. The control unit 211 sets a specified position in the virtual space that the user specifies by the posture, based on posture information detected by the controller 120, which is a posture detection means attached to the user's hand 405. Therefore, the image processing device 110 has a second setting means for setting a position in the virtual space based on the posture of the user. In the first embodiment, the specified position is set based on the posture of the controller 120, which is a hand controller. Then, the control unit 211 sets the position of the tip of the ray, the position, length, direction, etc. of the light beam based on the specified position. FIG. 6 shows an example of a ray display. A ray 420 is displayed from the controller 120 attached to the user's hand 405 toward the area in which the virtual object 403 exists.

[0043] The control unit 211 sets the initial position of the tip of the ray to a predetermined position within the operable range. In the first embodiment, the operable range is set to an area in which a gaze-determined virtual object exists, so the control unit 211 sets the initial position of the tip of the ray to a predetermined position such as the center position or the top end of the gazed-at virtual object. Note that the initial position of the tip of the ray is not limited to this, and may be a position determined for each virtual object, for example.

[0044] Furthermore, the control unit 211 restricts the movement of the tip position of the ray to within the operable range set in step S1003. In the example of FIG. 6, the control unit 211 restricts the tip position of the ray to be movable only within the area in which the gaze-determined virtual object 403 exists. Therefore, the image processing device 110 has a second setting means for restricting a position to be set in the virtual space based on the user's posture after an area (operable range) is set in the virtual space based on the user's line of sight to within the area. Specifically, it is assumed that the user's designated position in the virtual space, which is set based on the posture information detected by the controller 120, has gone out of the operable range after the operable range was set in step S1003. In this case, the control unit 211 sets a restricted position that is a position restricted within the operable range, and generates an image such that an image indicating the position that the user is operating (the tip of the ray) is displayed at the restricted position. On the other hand, when the user's designated position in the virtual space, which is set based on the posture information detected by the controller 120, is within the operable range, the control unit 211 generates an image so that an image (the tip of the ray) indicating the position that the user is operating is displayed at the designated position.

[0045] Furthermore, the control unit 211 generates an image of the virtual space so that an image indicating the operable range is displayed. In the example shown in Fig. 6, the operable range 430 is displayed as a frame-shaped image formed of dashed lines around the virtual object 403 determined to be gazed upon. By displaying the image indicating the operable range, the user can know that the position to which the tip of the ray 420 can be moved using the controller 120 is limited to within the operable range.

[0046] Furthermore, after setting the operable range in step S1003, control unit 211 generates an image of the virtual space so as not to display information (gaze cursor) indicating the gaze position set based on the gaze information detected by gaze detection unit 205. In the example shown in Fig. 6, gaze cursor 410 displayed in Fig. 5 is not displayed.

[0047] In step S1005, the control unit 211 causes the gaze detection unit 205 to detect the gaze information of the user.

[0048] In step S1006, the control unit 211 determines whether the user's gaze position in the virtual space set based on the user's gaze information detected in step S1005 is outside the operable range. If it is determined that the user's gaze position is outside the operable range (step S1006: YES), the control unit 211 executes the process of step S1007. When it is determined that the position is within the operable range (step S1006: NO), the control unit 211 executes the process of step S1008.

[0049] In step S1007, the control unit 211 displays the gaze cursor that was hidden in step S1004 again. FIG. 7 shows a display example in the case where the gaze position moves outside the operable range and onto the virtual object 402 after the operable range is set. As shown in FIG. 7, the gaze cursor that was hidden in FIG. 6 is displayed as the gaze cursor 411 in FIG. 7. The control unit 211 may differentiate the gaze cursor that is displayed when the gaze position of the user moves outside the operable range after the operable range is set from the gaze cursor that is displayed in the state where the operable range is not set. In the example of FIG. 7, the gaze cursor 411 is an image that is made up of an outer frame and a center point made up of dotted lines, and is different from the gaze cursor 410 in FIG. 5 that is made up of an outer frame and a center point made up of solid lines. In this way, by displaying the gaze cursor in a different manner, when the gaze position moves outside the operable range after the operable range is set, the user can be notified of this.

[0050] In step S1008, the control unit 211 maintains the state in which the gaze cursor is not displayed in step S1004.

[0051] In step S1009, the control unit 211 sets the display of an image (ray) indicating the position that the user is operating in the virtual space, based on the position information and attitude information of the controller 120 and the operable range. The control unit 211 controls the display of an image (ray) indicating the position that the user is operating in the virtual space, depending on whether or not the designated position in the virtual space based on the attitude of the user set based on the attitude information of the controller 120 is outside the operable range. Specifically, when the designated position based on the attitude of the user is within the operable range, the control unit 211 displays the tip of the ray at the designated position, and when the designated position is outside the operable range, the control unit 211 displays the tip of the ray at a restricted position that is restricted within the operable range. The restricted position is, for example, the position closest to the designated position on the boundary line of the operable range. As a result, even if the user performs an operation using the controller 120 that causes the tip of the ray to deviate from the operable range, the tip position of the ray is displayed so as not to move beyond the boundary of the operable range. A state in which the designated position is outside the operable range and the tip of the ray is displayed at the restricted position is called a "restricted state."

[0052] In step S1010, the control unit 211 determines whether the tip position of the ray is in a limited state. If it is determined that the tip position of the ray is in a limited state, the control unit 211 executes the process of step S1011, and if not, executes the process of step S1012.

[0053] In step S1011, the control unit 211 switches the display of the operable range to a warning display. In the example shown in FIG. 7, the image showing the operable range 431 is a frame-shaped image formed by dotted lines, which is different from the image showing the operable range 430 in the case of not being in the restricted state shown in FIG. 6. In addition, the tip position of the ray 421 is displayed within the operable range 431, but the starting point of the ray 421 is displayed at a position away from the controller 120. In addition, the control unit 211 makes the image showing the ray in the restricted state different from the image showing the ray in the not restricted state. In the example shown in FIG. 7, the ray 421 in the restricted state is displayed by a broken line, which is different from the ray 420 in the normal state not in the restricted state shown in FIG. 6. In this way, after the operable range is set, the image showing the operable range and the image of the ray are made different between when the designated position is outside the operable range and when it is within the operable range, so that the user can be notified that the designated position is outside the operable range.

[0054] The method of displaying the ray in the restricted state is not limited to this. For example, as shown in FIG. 8, the position of the origin of the ray may be the controller 120, and the ray 422 may be displayed in a curved shape. stomach.

[0055] In step S1012, the control unit 211 changes the display of the operable range to a normal display, for example, the operable range 430 shown in FIG.

[0056] In step S1013, the control unit 211 determines whether a state in which the user's gaze position is outside the operable range (condition A) and the user's designated position is outside the operable range (condition B) has continued for a predetermined time. Condition B can also be said to be a state in which the tip position of the ray is restricted. The method for determining condition A is the same as the method for determining condition B, and the method for determining condition B is the same as the method for determining condition S1010. If it is determined that condition A and condition B have continued for the predetermined time, the control unit 211 executes the process of step S1014, and if not, returns to step S1005 and repeats the process.

[0057] In step S1014, the control unit 211 cancels the operable range set in step S1003, and generates an image of the virtual space in which an image (the tip of the ray) indicating the position that the user is operating is displayed at a position specified by the user's posture. In the first embodiment, the control unit 211 hides the operable range 430 and the ray 420 shown in Fig. 6, and displays the gaze cursor 410 shown in Fig. 5. This results in the display shown in Fig. 5.

[0058] According to the first embodiment, after the virtual object 403 is selected, the operation range using the controller 120 can be limited to an operable range set in an area where the virtual object 403 exists. Therefore, the user does not need to perform an operation using the controller 120 to keep the tip position of the ray within the area where the virtual object to be operated exists. This reduces the burden on the user.

[0059] In the first embodiment, an example is shown in which a virtual object present at the user's gaze position is selected and an operable range is set based on the selected virtual object, but the method of setting the operable range is not limited to this. For example, a predetermined range centered on the user's gaze position may be set as the operable range. In addition, the method of setting the operable range based on a virtual object is not limited to the above-mentioned method of setting the area where the virtual object exists as the operable range. For example, an area including a predetermined range around the area where the virtual object exists may be set as the operable range, or a part of the area where the virtual object exists may be set as the operable range.

[0060] <Variation 1> In the first modification, an example in which the condition for gaze determination in step S1002 in FIG. 3 is changed will be described. In the first embodiment, when the gaze position of the user in the virtual space set based on the gaze information detected by the gaze detection unit 205 exists within a certain range for a predetermined time, it is determined that the user is gazing at the gaze position within the range. Also, when the gaze position of the user is within a range in which a certain virtual object exists for a predetermined time, it is determined that the user is gazing at the virtual object. In the first modification, further, when the designated position of the user in the virtual space set based on the posture information detected by the controller 120 exists within a certain range for a predetermined time, it is determined that the user is gazing at the gaze position within the range. That is, when the gaze position and the designated position are within a certain range in the virtual space for a predetermined time, the control unit 211 of the image processing device 110 sets the gaze position within the range. Also, when the gaze position and the designated position of the user are within a range in which a certain virtual object exists for a predetermined time, the control unit 211 determines that the user is gazing at the virtual object.

[0061] In FIG. 9, in addition to the gaze cursor 410 shown in FIG. In the example shown in FIG. 10, a ray 420 is displayed toward the inside of the area of ​​the virtual object 403. If the state in which the gaze cursor 410 and the tips of the ray 420 are present on the virtual object 403 continues for a predetermined time, the control unit 211 performs a gaze determination on the virtual object 403, and executes the process of step S1003. At this time, the predetermined time (determination threshold) may be shorter than that in the first embodiment. When performing such a gaze determination, the control unit 211 displays the gaze cursor 410 and the ray 420 in the image display area 401 at the time of step S1001.

[0062] Also, the gaze determination may be made based only on the posture information detected by the controller 120, without using the gaze information detected by the gaze detection unit 205. In this case, the control unit 211 sets a designated position based on the posture of the user in the virtual space based on the posture information by the controller 120, and selects an object arranged in the virtual space based on the designated position. Then, the control unit 211 sets an operable range in the virtual space based on the information of the selected object. In this case, the image processing device 110 has a second setting means for setting a position in the virtual space based on the posture of the user viewing the virtual space, and a selection means for selecting an object arranged in the virtual space based on the set position. Also, the image processing device 110 has a third setting means for setting an area in the virtual space based on the information of the object. Note that the control unit 211 can set the operable range based on the shape information of the object, as in the first embodiment. Also, the control unit 211 can limit the designated position based on the posture of the user to within the operable range after the operable range is set. Also, the control unit 211 can control not to display information (gaze cursor) indicating the position of the user's gaze in the virtual space after the operable range is set. Furthermore, the control unit 211 can control so that, when the user's line of sight goes outside the operable range after the operable range is set, information indicating the position of the user's line of sight (gaze cursor) is displayed again in the virtual space. Furthermore, the control unit 211 can notify the user by, for example, changing the drawing mode of the gaze cursor when the user's line of sight goes outside the operable range after the operable range is set. Furthermore, the control unit 211 can notify the user by, for example, changing an image or a ray image indicating the operable range when a position in the virtual space designated based on the user's posture goes outside the operable range after the operable range is set. Furthermore, the control unit 211 can cancel the setting of the operable range when a state in which a designated position based on the user's posture is outside the operable range and the user's line of sight is outside the operable range continues for a predetermined time after the operable range is set.When a designated position based on the user's posture, which is set based on posture information detected by the controller 120, exists within a certain range for a predetermined period of time, the control unit 211 can determine that the user is gazing at a gaze position within the range. Furthermore, when a state in which a designated position based on the user's posture exists within a range in which a certain virtual object exists continues for a predetermined period of time, the control unit 211 can determine that the user is gazing at the virtual object. Furthermore, when a button mounted on the controller 120 is operated while a designated position based on the user's posture exists within a range in which a certain virtual object exists, the control unit 211 can determine that the user is gazing at the virtual object.

[0063] <Variation 2> In the first embodiment and the first variant example, examples were described in which the display of the ray was controlled based on the position information and posture information of the controller 120. In the second variant example, a method will be described in which a user's hand is detected from a captured image and the ray is controlled based on information about the detected user's hand.

[0064] 10 shows a state in which a ray 420 is displayed from the fingertip of a user's hand 406 toward the area of ​​a virtual object 403. The control unit 211 detects the user's hand shown in the image captured by the imaging unit 202, and acquires position information and posture information of the user's hand. Next, the control unit 211 sets the display position and direction of the ray based on the position information and posture information of the user's hand, and displays the ray 420 within the image display area 401.

[0065] The control of the ray 420 based on the position information and orientation information of the controller 120 described in the first embodiment and the first modified example can be replaced with control of the ray 420 based on the position information and orientation information of the user's hand 406 .

[0066] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0067] <Other Examples> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. Also, the present invention can be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0068] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A first setting means for setting an area in the virtual space based on a line of sight of a user viewing the virtual space; A second setting means for setting a position in the virtual space based on a posture of the user; having The information processing apparatus according to claim 1, wherein the second setting means, after the area is set by the first setting means, limits a setting position to within the area. (Configuration 2) a selection means for selecting an object arranged in the virtual space based on a line of sight of the user; 2. The information processing apparatus according to configuration 1, wherein the first setting means sets the area based on information about the object. (Configuration 3) 3. The information processing apparatus according to configuration 2, wherein the first setting means sets the area based on shape information of the object. (Configuration 4) 4. The information processing device according to any one of configurations 1 to 3, wherein after the area is set by the first setting means, information indicating a position of the user's line of sight is not displayed within the virtual space. (Configuration 5) An information processing device according to configuration 4, further comprising: if the user's gaze moves outside the area after the area is set by the first setting means, information indicating the position of the user's gaze within the virtual space is displayed again. (Configuration 6) 6. The information processing device according to any one of configurations 1 to 5, wherein after the area is set by the first setting means, when the user's line of sight moves outside the area, the user is notified of this. (Configuration 7) An information processing device according to any one of configurations 1 to 6, which notifies a user when a position in the virtual space specified based on the user's posture goes outside the area after the area is set by the first setting means. (Configuration 8) An information processing device as described in any one of configurations 1 to 7, wherein after the area is set by the first setting means, if a position in the virtual space specified based on the user's posture is outside the area and the user's line of sight is outside the area for a predetermined period of time, the first setting means cancels the setting of the area. (Configuration 9) A second setting means for setting a position in the virtual space based on a posture of the user viewing the virtual space; a selection means for selecting an object arranged in the virtual space based on the position set by the second setting means; a third setting means for setting an area in the virtual space based on information about the object; having The information processing apparatus according to claim 1, wherein the second setting means, after the area is set by the third setting means, limits a setting position to within the area. (Configuration 10) 10. The information processing apparatus according to configuration 9, wherein the third setting means sets the area based on shape information of the object. (Configuration 11) 11. The information processing device according to configuration 9 or 10, wherein after the area is set by the third setting means, information indicating the position of the user's line of sight is not displayed within the virtual space. (Configuration 12) An information processing device according to configuration 11, wherein after the area is set by the third setting means, if the user's line of sight moves outside the area, information indicating the position of the user's line of sight is displayed again within the virtual space. (Configuration 13) The information processing device according to any one of configurations 9 to 12, further comprising: a notification to a user when the user's line of sight moves outside the area after the area is set by the third setting means (Configuration 14). An information processing device as described in any one of configurations 9 to 13, which notifies the user if a position in the virtual space specified based on the user's posture goes outside the area after the area is set by the third setting means. (Configuration 15) An information processing device as described in any one of configurations 9 to 14, wherein after the area is set by the third setting means, if a position in the virtual space specified based on the user's posture is outside the area and the user's line of sight is outside the area for a predetermined period of time, the third setting means cancels the setting of the area. (Method 1) A first setting step of setting an area in the virtual space based on a line of sight of a user viewing the virtual space; A second setting step of setting a position in the virtual space based on the posture of the user; having The information processing method, wherein in the second setting step, after the area is set in the first setting step, the setting position is restricted to within the area. (Method 2) A second setting step of setting a position in the virtual space based on a posture of the user viewing the virtual space; a selection step of selecting an object arranged in the virtual space based on the position set in the second setting step; a third setting step of setting an area in the virtual space based on information about the object; having The information processing method, wherein in the second setting step, after the area is set in the third setting step, the setting position is restricted to within the area. (program) A computer functions as each of the means of the information processing device according to any one of configurations 1 to 15. A program to make it work. [Explanation of symbols]

[0069] 1: information processing system, 110: image processing device, 205: gaze detection unit, 211: control unit, 224: movement detection unit

Claims

1. A first setting means for setting an area in the virtual space based on a line of sight of a user viewing the virtual space; A second setting means for setting a position in the virtual space based on a posture of the user; having The information processing apparatus according to claim 1, wherein the second setting means, after the area is set by the first setting means, limits a position to be set within the area.

2. a selection means for selecting an object arranged in the virtual space based on a line of sight of the user; The information processing apparatus according to claim 1 , wherein the first setting means sets the area based on information about the object.

3. The information processing apparatus according to claim 2 , wherein the first setting means sets the area based on shape information of the object.

4. The information processing device according to any one of claims 1 to 3, wherein after the area is set by the first setting means, information indicating the position of the user's line of sight is not displayed within the virtual space.

5. The information processing device according to claim 4 , wherein, after the area is set by the first setting means, when the user's line of sight moves outside the area, information indicating the position of the user's line of sight is displayed again within the virtual space.

6. 4. The information processing device according to claim 1, further comprising: a display unit configured to display a display area including a display region that displays a display area including a display region that displays a display area.

7. An information processing device according to any one of claims 1 to 3, which notifies the user when a position in the virtual space specified based on the user's posture goes outside the area after the area is set by the first setting means.

8. An information processing device as described in any one of claims 1 to 3, wherein after the area is set by the first setting means, if a position in the virtual space specified based on the user's posture is outside the area and the user's line of sight is outside the area for a predetermined period of time, the first setting means cancels the setting of the area.

9. A second setting means for setting a position in the virtual space based on a posture of the user viewing the virtual space; a selection means for selecting an object arranged in the virtual space based on the position set by the second setting means; a third setting means for setting an area in the virtual space based on information about the object; having The information processing apparatus according to claim 1, wherein the second setting means, after the area is set by the third setting means, limits the setting position to within the area.

10. The information processing apparatus according to claim 9 , wherein the third setting means sets the area based on shape information of the object.

11. After the area is set by the third setting means, the user's visual field is set in the virtual space.

11. The information processing device according to claim 9, wherein information indicating the position of the line is not displayed.

12. The information processing device according to claim 11 , wherein, after the area is set by the third setting means, when the user's line of sight moves outside the area, information indicating the position of the user's line of sight is displayed again within the virtual space.

13. The information processing device according to claim 9 or 10, wherein, after the area is set by the third setting means, when the user's line of sight moves outside the area, the user is notified of this.

14. 11. The information processing device according to claim 9, wherein, after the area is set by the third setting means, if a position in the virtual space specified based on the user's posture goes outside the area, the user is notified of this.

15. The information processing device of claim 9 or 10, wherein after the area is set by the third setting means, if a position in the virtual space specified based on the user's posture is outside the area and the user's line of sight is outside the area for a predetermined period of time, the third setting means cancels the setting of the area.

16. A first setting step of setting an area in the virtual space based on a line of sight of a user viewing the virtual space; a second setting step of setting a position in the virtual space based on the posture of the user; having In the second setting step, after the area is set in the first setting step, the setting position is restricted to within the area.

17. a second setting step of setting a position in the virtual space based on a posture of the user viewing the virtual space; a selection step of selecting an object arranged in the virtual space based on the position set in the second setting step; a third setting step of setting an area in the virtual space based on information about the object; having The information processing method according to the present invention, wherein in the second setting step, after the area is set in the third setting step, the setting position is restricted to within the area.

18. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 3, 9 and 10.

Images