Information processing apparatus and information processing method

JP2024060939A5Pending Publication Date: 2025-10-20CANON KK

View PDF 0 Cites 0 Cited by

Patent Information

Application Number: JP2022168535
Authority / Receiving Office: JP · JP
Patent Type: Applications
Current Assignee / Owner
Filing Date: 2022-10-20
Publication Date: 2025-10-20

AI Technical Summary

Technical Problem

Users face difficulty in selecting a desired virtual object from a plurality of virtual objects arranged in a three-dimensional space.

Method used

An information processing device that includes a display control unit to place virtual objects in a user's field of view and an operating body at the user's hand position for selecting virtual objects within a specified range in three-dimensional space.

Benefits of technology

Enables easy selection of desired virtual objects in a three-dimensional space without visual discrepancies.

✦ Generated by Eureka AI based on patent content.

Smart Images

Figure 00000000_0000_ABST

Patent Text Reader

Abstract

To provide an information processing apparatus configured to allow a user to easily select a desired virtual object from among a plurality of virtual objects arranged in a three-dimensional space.SOLUTION: An information processing apparatus includes: display control means which performs display control of a virtual object so as to be arranged in a three-dimensional space which is a field of view of a user; and selection means which puts the virtual object which is selected by an operator located in a position of a hand of the user and included in a selection range in the three-dimensional space, into a selected state.SELECTED DRAWING: Figure 1

Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, a Virtual Reality (VR) system, a Mixed Reality (MR) system, and an Augmented Reality (AR) system that combine real space and virtual space have been developed. For example, Patent Document 1 proposes a method of depicting a real object (physical item) selected by a user in an AR scene generated by a computer. Patent Document 2 proposes a method of selecting an enemy character by encircling it with a predetermined trajectory shape using a pen, and registering it as an attack target. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-125258 [Patent Document 2] JP 2010-17395 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, it has been difficult for a user to select a desired virtual object from among a plurality of virtual objects arranged in the depth direction in a three-dimensional virtual space.

[0005] An object of the present invention is to provide an information processing device that enables a user to easily select a desired virtual object from a plurality of virtual objects arranged in a three-dimensional space. [Means for solving the problem]

[0006] The information processing device according to the present invention is characterized by having a display control means for controlling the display of a virtual object so that it is positioned within a three-dimensional space within the user's field of vision, and a selection means for selecting the virtual object included in a selection range in the three-dimensional space selected using an operating object at the position of the user's hand. Effect of the Invention

[0007] According to the present invention, a user can easily select a desired virtual object from a plurality of virtual objects placed in a three-dimensional space. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a block diagram illustrating an example of a functional configuration of the image processing system. [Diagram 2] 4 is a flowchart showing a process of the information processing device according to the first embodiment. [Diagram 3] 11A and 11B are diagrams illustrating an example of an operation for changing a selection state of a virtual object. [Figure 4] FIG. 4 is a diagram illustrating a selection range according to the first embodiment. [Diagram 5] FIG. 11 is a diagram illustrating another example of the selection range according to the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating a selection range according to a second embodiment. [Figure 7] FIG. 13 is a diagram illustrating a selection range according to the third embodiment. [Figure 8] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The embodiment described below is an example of a means for realizing the present invention, and can be appropriately changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. In addition, it is also possible to appropriately combine parts of the following embodiments.

[0010] [First embodiment] 1 is a block diagram showing an example of a functional configuration of an image processing system 100 according to this embodiment. The image processing system 100 is a system for presenting a mixed reality space (MR space) that combines a real space and a virtual space to a person (user) experiencing the system. The image processing system 100 performs display control so that a virtual object is placed in a three-dimensional space that is within the user's field of vision. The user can select a desired virtual object by specifying (selecting) a selection range in the three-dimensional space.

[0011] In this embodiment, it is assumed that an MR space is presented to a user by displaying a composite image obtained by combining an image of a real space with an image of a virtual space rendered by computer graphics (CG).

[0012] The image processing system 100 includes a display device 1000 and an information processing device 1100. The information processing device 1100 can synthesize an image of real space captured from the display device 1000 with an image of virtual space generated by the information processing device 1100, and output the synthesized image to the display device 1000 as a mixed reality image (MR image).

[0013] The image processing system 100 relates to a system that displays an image of a virtual space, and is not limited to an MR (mixed reality) system that displays an MR image that combines an image of a real space with an image of a virtual space. The image processing system 100 can also be applied to a VR (virtual reality) system that presents only an image of a virtual space to a user, or an AR (augmented reality) system that presents an image of a virtual space to a user by transmitting the real space.

[0014] The display device 1000 includes an imaging unit 1010. The imaging unit 1010 captures images of real space continuously in time series, and outputs the captured images of real space to the information processing device 1100. The imaging unit 1010 may include a stereo camera consisting of two cameras fixed to each other so as to capture images of real space in the line of sight direction from the viewpoint position of the user.

[0015] The display device 1000 includes a display unit 1020. The display unit 1020 displays the MR image output from the information processing device 1100. The display unit 1020 may include two displays arranged corresponding to the left and right eyes of the user, respectively. In this case, the left-eye display corresponding to the left eye of the user displays the MR image for the left eye, and the right-eye display corresponding to the right eye of the user displays the MR image for the right eye.

[0016] The display device 1000 is, for example, a head-mounted display device (HMD). However, the display device 1000 is not limited to an HMD and may be a handheld display (HHD). An HHD is a handheld display that a user holds in his or her hand and peers into like binoculars to observe images. Furthermore, the display device 1000 may be a display terminal such as a tablet or a smartphone.

[0017] The information processing device 1100 and the display device 1000 are connected to each other so as to be able to perform data communication with each other. The connection between the information processing device 1100 and the display device 1000 may be wired or wireless. Furthermore, the information processing device 1100 may be disposed inside the housing of the display device 1000.

[0018] The information processing device 1100 includes a position and orientation acquisition unit 1110 , a selection unit 1120 , an image generation unit 1130 , an image synthesis unit 1140 , and a data storage unit 1150 .

[0019] The position and orientation acquisition unit 1110 acquires the position and orientation of the imaging unit 1010 in the world coordinate system and the position of the observer's (user's) hand. Specifically, the position and orientation acquisition unit 1110 extracts markers assigned to the world coordinate system from an image of real space captured by the imaging unit 1010. The position and orientation acquisition unit 1110 acquires the position and orientation of the imaging unit 1010 in the world coordinate system based on the position and orientation of the markers, and outputs the acquired position and orientation information of the imaging unit 1010 to the data storage unit 1150.

[0020] The position and orientation acquisition unit 1110 extracts a feature region of the hand from an image of real space captured by the imaging unit 1010. The position and orientation acquisition unit 1110 acquires position information of each part of the hand using the extracted feature region of the hand and hand shape information stored in the data storage unit 1150. The position information of each part of the hand may be position information of a part of the hand such as the fingertips or joints of each finger. The position and orientation acquisition unit 1110 only needs to acquire position information of parts used by the selection unit 1120 to set a selection range. For example, when the user specifies a selection range with a fingertip, it is sufficient to acquire position information of the fingertip.

[0021] Note that the method of acquiring the position and orientation of the image capturing unit 1010 is not limited to the above method. For example, the position and orientation acquiring unit 1110 may perform Simultaneous Localization and Mapping (SLAM) processing based on feature points captured in an image to acquire the position and orientation of the image capturing unit 1010 and the position and orientation in an individual coordinate system.

[0022] Furthermore, the position and orientation of the imaging unit 1010 may be obtained by attaching a sensor whose relative position and orientation with respect to the imaging unit 1010 is known to the display device 1000 and using a measurement value from the sensor. The position and orientation acquisition unit 1110 can obtain the position and orientation of the imaging unit 1010 in the world coordinate system by converting the measurement value from the sensor based on the position and orientation of the sensor relative to the imaging unit 1010. Furthermore, the position and orientation acquisition unit 1110 may obtain the position and orientation of the imaging unit 1010 by using a motion capture system.

[0023] Also, the method of acquiring the position of the user's hand is not limited to the above method. For example, when the imaging unit 1010 is a monocular camera, the position and orientation acquisition unit 1110 can acquire the position of the hand by a ToF (Time of flight) sensor. Also, when the imaging unit 1010 has multiple cameras with different positions and orientations, the position and orientation acquisition unit 1110 may acquire the position of the hand based on images captured by the multiple cameras. The position and orientation acquisition unit 1110 can obtain the depth of the entire image from the stereo image by a method such as SemiGlobalMatching (SGM), and acquire the position of the hand using the depth information and hand shape information.

[0024] The position and orientation acquisition unit 1110 may acquire the hand position by having the user wear gloves equipped with sensors capable of acquiring the positions of each joint of the hand. The position and orientation acquisition unit 1110 may acquire the hand position using a motion capture system. In addition, when the user performs an operation using an MR controller placed at the hand position, the position and orientation acquisition unit 1110 may acquire the position of the controller as the hand position.

[0025] The selection unit 1120 sets a selection range in the virtual space based on the position of the user's hand acquired by the position and orientation acquisition unit 1110. The selection unit 1120 changes the selection state of the virtual space data (virtual object) included in the selection range.

[0026] The image generating unit 1130 generates an image based on the virtual space data stored in the data storage unit 1150. The data of the virtual space includes data on each virtual object constituting the virtual space, data on realistic approximation objects obtained by importing three-dimensional shape information of real objects obtained from the real space into the virtual space, and data on a light source that illuminates the virtual space.

[0027] The image generating unit 1130 sets a virtual viewpoint based on the position and orientation of the imaging unit 1010 acquired by the position and orientation acquiring unit 1110. The image generating unit 1130 can set, for example, the position of the user's dominant eye or the midpoint between the left and right eyes as the virtual viewpoint. The user's dominant eye can be set in advance. When the imaging unit 1010 has multiple cameras, the image generating unit 1130 can set the virtual viewpoint based on the positional relationship between the positions of the respective cameras when the user is wearing the HMD (display device 1000) and the positions of the user's eyes.

[0028] The image generating unit 1130 generates an image of the virtual space seen from the set viewpoint (virtual space image). Note that since the technology for generating an image of the virtual space seen from a viewpoint having a predetermined position and orientation is well known, a detailed description thereof will be omitted.

[0029] The image synthesis unit 1140 synthesizes the image of the virtual space generated by the image generation unit 1130 and the image of the real space captured by the imaging unit 1010 to generate an MR image, which is an image of a three-dimensional space. The image synthesis unit 1140 outputs the generated MR image to the display unit 1020.

[0030] The data storage unit 1150 includes a RAM, a hard disk drive device, etc., and stores the various pieces of information described above. The data storage unit 1150 also stores information that will be described as known information and various types of setting information.

[0031] FIG. 2 is a flowchart showing an example of a process in which the information processing device 1100 generates an MR image and outputs it to the display device 1000.

[0032] In step S2010, the position and orientation acquisition unit 1110 acquires the position and orientation of the imaging unit 1010 and the position of the user's hand. The position of the user's hand includes information such as the positions of each part of the hand (e.g., fingertips, joints, etc.) and the position of a controller worn or held on the hand. Of this information, the position and orientation acquisition unit 1110 only needs to acquire information used to set a selection range.

[0033] In step S2020, the selection unit 1120 determines whether or not the mode is a mode for selecting a virtual object (selection mode). A method for determining whether or not the mode has shifted to the selection mode will be described later with reference to FIG. 3. If it is determined that the mode is the selection mode, the process proceeds to step S2030. If it is determined that the mode is not the selection mode, the process proceeds to step S2050.

[0034] In step S2030, the selection unit 1120 updates a selection range for changing the selection state of a virtual object that is arranged and displayed in a virtual space, based on the position of the user's hand. The selection range is a three-dimensional range obtained by expanding in the depth direction a two-dimensional selection area specified using a manipulation object at the position of the user's hand. The manipulation object may be the user's hand or a controller. In the following description, the manipulation object is the user's hand.

[0035] In step S2040, the selection unit 1120 updates the selection state of the virtual objects included in the selection range based on the updated selection range. The selection unit 1120 sets the virtual objects included in the selection range to a selected state, and sets the virtual objects not included in the selection range to a deselected state. Note that the virtual objects included in the selection range may be virtual objects whose entirety is included in the selection range, or may be virtual objects whose part (a predetermined percentage or more) is included. Furthermore, the virtual objects included in the selection range may be virtual objects whose center of gravity is included in the selection range.

[0036] In step S2050, the image generating unit 1130 generates an image of the virtual space seen from the virtual viewpoint, using the information on the position and orientation of the imaging unit 1010 acquired in step S2010.

[0037] In step S2060, the image synthesis unit 1140 synthesizes the image of the virtual space generated in step S2050 with the image of the real space captured by the imaging unit 1010 (real space image) to generate an MR image.

[0038] In step S2070, the information processing device 1100 determines whether or not the termination condition is satisfied. For example, the information processing device 1100 can determine that the termination condition is satisfied when an instruction to terminate the process of generating an MR image is input. If the termination condition is satisfied, the information processing device 1100 terminates the process shown in Fig. 2. If the termination condition is not satisfied, the information processing device 1100 returns to the process of step S2010.

[0039] Fig. 3 is a diagram for explaining an example of an operation for changing the selection state of a virtual object. Fig. 3(A) to 3(F) show examples of a composite image generated by the image composition unit 1140. Fig. 3(A) to 3(F) also show an example in which the selection unit 1120 sets a selection range in a virtual space and changes the selection state of a virtual object.

[0040] The real space image 3010 is an image representing the real space, and is an image captured by the imaging unit 1010. The real space image 3010 is synthesized with an image of a virtual space in which various objects are arranged.

[0041] In a composite image obtained by combining a real space image 3010 and a virtual space image, objects representing the user's left hand 3020 and right hand 3030 are arranged. In addition, objects representing virtual objects 3040, 3050, 3060, and 3070 are arranged in the composite image. Coordinate axes 3080 indicate a coordinate system in the virtual space. The X-axis direction corresponds to the horizontal direction of the MR image displayed on the display. The Y-axis direction corresponds to the vertical direction of the MR image displayed on the display. The Z-axis direction corresponds to the depth direction of the virtual space.

[0042] The composite image may display a hand trajectory 3090 for setting a selection range. The composite image may also display a selection region 3100. The selection region 3100 is a two-dimensional region surrounded by a trajectory 3090 of the user's hand position or fingertip position. A three-dimensional selection range in the virtual space is set by expanding the selection region 3100 in the depth direction.

[0043] 3A shows a state where the selection state of a virtual object in a virtual space has not been changed. The image generating unit 1130 generates a virtual space image based on the position and orientation of the imaging unit 1010 acquired by the position and orientation acquiring unit 1110. The image synthesizing unit 1140 synthesizes the virtual space image with the real space image captured by the imaging unit 1010 to generate an MR image (synthetic image).

[0044] Fig. 3(B) shows a state in which a selection mode for changing the selection state of a virtual object starts. In the example of Fig. 3(B), the selection mode starts when the distance between the user's left hand 3020 and right hand 3030 becomes equal to or smaller than a threshold.

[0045] The position and orientation acquisition unit 1110 acquires the positions of the user's left hand 3020 and right hand 3030. In the example of FIG. 3(B), the position and orientation acquisition unit 1110 may acquire the position of the index fingertip as the hand position. The selection unit 1120 sets a selection area 3100 based on the positions of the user's left hand 3020 and right hand 3030. The selection unit 1120 sets a selection range in virtual space based on the selection area 3100, and determines the selection state of a virtual object included in the selection range. Change the state.

[0046] The condition for starting the selection mode may be that the distance between the left hand and the right hand is equal to or less than a threshold for a predetermined time. The selection mode may be started by a user operation from an input device such as a button. The selection mode may also be started by a user's gaze or voice command.

[0047] FIG. 3(C) shows a hand trajectory 3090 when the user's left hand 3020 and right hand 3030 move in directions away from each other from the state shown in FIG. 3(B).

[0048] Fig. 3(D) shows a state in which the user's left hand 3020 and right hand 3030 have further moved from the state in Fig. 3(C) to surround the virtual object, and the distance between them has again become equal to or less than the threshold. The position and orientation acquisition unit 1110 acquires the positions of the user's left hand 3020 and right hand 3030, and the selection unit 1120 sets a selection area 3100 based on the acquired hand position information. The selection unit 1120 sets the selection area 3100 and ends the selection mode.

[0049] The condition for ending the selection mode may be, for example, that the distance between the left and right hands is equal to or less than a threshold. The selection mode may also be ended by a change in the shape of the hand, such as bending the fingertips. The selection mode may also be ended by a user operation from an input device such as a button, or by a user's gaze or voice instruction.

[0050] 3(D), virtual object 3040 and virtual object 3070 are included in a selection range obtained by expanding selection region 3100 in the depth direction, and selection unit 1120 changes virtual object 3040 and virtual object 3070 to a selected state. Virtual object 3040 and virtual object 3070 in a selected state may be displayed in an emphasized manner, such as by making their outlines thicker than other virtual objects or by highlighting them. Note that a method for displaying a virtual object in a selected state may be any display mode different from that in a non-selected state (deselected state), and may be a wireframe display, or may be displayed in a color or transparency different from that in the deselected state.

[0051] Whether or not a virtual object is included in the selection range is not limited to being determined based on whether or not the entire virtual object is included in the selection range. Whether or not a virtual object is included in the selection range may be determined based on whether the center of gravity of the virtual object is included in the selection range, whether the entire virtual object (each vertex of a rectangular parallelepiped circumscribing the virtual object) is included in the selection range, or whether a predetermined percentage or more of the virtual object is included in the selection range.

[0052] 3(A) to 3(D) show an example in which the user specifies the selection area 3100 using both hands, but the selection area 3100 may be specified with one hand. For example, the user may determine the positions of the start point and end point of the selection area 3100 by changing the shape of one hand into a predetermined shape, and specify the area surrounded by the trajectory 3090 of the hand from the start point to the end point as the selection area 3100.

[0053] Fig. 4 is a diagram for explaining the selection range according to the first embodiment. Specifically, a method for determining the coordinates of the selection range set in a virtual space will be explained. The selection range is a three-dimensional range obtained by expanding the two-dimensional selection area 3100 explained in Fig. 3(D) in the depth direction. In Fig. 4, the coordinates of the selection range are explained on a two-dimensional plane consisting of the Z axis in the depth direction and the X axis in the horizontal direction in a three-dimensional space consisting of the XYZ axes.

[0054] The origin 4010 of the observation direction corresponds to the position of the user, specifically, the position where the user is observing the virtual space. The origin 4010 can be the position of either the left or right eye of the user, or the position of the user's dominant eye. The origin 4010 can be the position of the camera if that position is approximately the same as the user's eye position.

[0055] In addition, when the image capturing unit 1010 of the display device 1000 includes a plurality of cameras, and the positions of the cameras and the user's eyes are different, the origin 4010 can be the position of the user's viewpoint (virtual viewpoint) converted from the camera position. The position of the virtual viewpoint may be the position of the dominant eye, or the midpoint between the left and right eyes.

[0056] The coordinates of the origin 4010 are (X0, Y0, Z0). The Z coordinate in the depth direction of the user's left hand 3020 and right hand 3030 is ZH. In other words, the Z direction coordinate of the two-dimensional selection area 3100 enclosed by the trajectory of the user's hands is ZH.

[0057] Selection area 3100 may be an area surrounded by the trajectory 3090 of the user's hand that approximates a rectangle, an ellipse, or the like. In this way, by setting selection area 3100 by approximating the trajectory 3090 actually specified by the user to a shape such as a rectangle or a circle, the amount of calculation required to find the coordinates of the three-dimensional selection range is reduced.

[0058] In the following description, the selection area 3100 will be described as a rectangle. For example, the coordinates of one vertex can be defined as (XHS, YHS, ZH), and the coordinates of the diagonal vertex can be defined as (XHE, YHE, ZH). In other words, the selection area 3100 is an area surrounded by the four points (XHS, YHS, ZH), (XHS, YHE, ZH), (XHE, YHE, ZH), and (XHE, YHS, ZH).

[0059] The selection unit 1120 expands the selection region 3100 in the Z-axis direction, which is the depth direction. If a rectangular parallelepiped in which only the Z-axis coordinate of the selection region 3100 is changed is formed as the selection range, the range of the formed rectangular parallelepiped may be shifted from the range of the selection region 3100 viewed by the user. Therefore, the selection unit 1120 expands the selection region 3100 in the depth direction based on the position of the user and the position on the contour of the selection region 3100 so that no apparent shift occurs.

[0060] The following calculation example shows how to obtain the selection range when the coordinate in the Z-axis direction of the center of gravity of virtual object 3040 is Z1. The distance in the Z direction from the user's position to the hand is ZH-Z0. The distance in the Z direction from the user's position to the center of gravity of virtual object 3040 is (Z1-Z0) / (ZH-Z0) times the distance to the hand. Here, (Z1-Z0) / (ZH-Z0)=k.

[0061] The vertex at Z coordinate Z1 corresponding to one vertex (XHS, YHS, ZH) of the two-dimensional selection area 3100 is (k×(XHS-X0)+X0, k×(YHS-Y0)+Y0, Z1). Similarly, the vertex at Z coordinate Z1 corresponding to the other vertex (XHE, YHE, ZH) is (k×(XHE-X0)+X0, k×(YHE-Y0)+Y0, Z1). In this way, by adjusting the X coordinate and Y coordinate according to the position of the virtual object in the depth direction, the selection unit 1120 can appropriately set a three-dimensional selection range in line with the user's intention.

[0062] Fig. 5 is a diagram for explaining another example of the selection range according to the first embodiment. In Fig. 4, the coordinates of the selection range in the Z coordinate Z1 are calculated using the coordinates of the positions on the contour of the selection area 3100 (the positions of the vertices if the selection area 3100 is a rectangle), but they may be calculated using the angle of view of the two-dimensional selection area 3100 without using ZH. That is, the selection unit 1120 sets the selection range using an origin 4010 corresponding to the position of the user and a range of directions from the origin 4010 to which the hand (operation object) is pointed. The method of using the range of directions to which the hand is pointed is effective when it is difficult to measure the distance to the hand.

[0063] A method for determining the selection range using the angle of view of the selection region 3100 will be described with reference to Fig. 5. Fig. 5 illustrates the selection range on the XZ plane consisting of the X axis and the Z axis, but the selection range on the YZ plane can also be shown by replacing the X axis with the Y axis and the angle θx with the angle θy.

[0064] The angle of view of the space viewed by the user is determined by the angle of view (θx, θy) of the camera mounted on the HMD with respect to the direction in which the user is looking. Within the angle of view of the camera, the angle that a line passing through one vertex of the selection area 3100 approximated to a rectangle and the origin 4010 makes with a line passing through the origin 4010 and parallel to the Z axis is defined as (θxs, θys). In addition, the angle that a line passing through the other diagonal vertex and the origin 4010 makes with a line passing through the origin 4010 and parallel to the Z axis is defined as (θxe, θye).

[0065] For example, at a distance of Z coordinate Z1, the coordinates (XVS1, YVS1, Z1) and (XVE1, YVE1, Z1) of the ends of the camera's angle of view can be calculated using the camera's angle of view by the following formula. XVS1=X0+(Z1-Z0)×tan-θx YVS1=Y0+(Z1-Z0)×tan-θy XVE1 = X0 + (Z1 - Z0) × tan θx YVE1 = Y0 + (Z1 - Z0) × tan θx

[0066] Using the same calculation method, the coordinates (XS1, YS1, Z1) of one vertex of the selection range at a distance of Z coordinate Z1 can be calculated using the following formula. XS1=XVS1+(Z1-Z0)×(tanθx-tanθxs) YS1=YVS1+(Z1-Z0)×(tanθy-tanθys)

[0067] Similarly, the coordinates of the other vertex (XE1, YE1, Z1) can be calculated using the following formula. XE1=XVE1-(Z1-Z0)×(tanθx-tanθxe) YE1=YVE1-(Z1-Z0)×(tanθy-tanθye)

[0068] If the selected region 3100 is not rectangular, the selection unit 1120 can reduce the amount of calculations by approximating the selected region 3100 to a rectangle.

[0069] In the above-described first embodiment, the information processing device 1100 sets the selection range by expanding in the depth direction the selection region 3100 specified by using an operating object such as the user's hand or a controller. Specifically, the information processing device 1100 can set the selection range by expanding the selection region 3100 in the depth direction using an origin corresponding to the position of the user and a position on the contour of the selection region 3100.

[0070] Furthermore, the information processing device 1100 sets the selection range using an origin corresponding to the position of the user and a range of directions in which the operating object is directed from the origin. Specifically, the information processing device 1100 can set the selection range using an angle formed between the Z axis and a straight line passing through the origin 4010 and a point on the contour of the selection region 3100. Note that the contour of the selection region 3100 may be approximated to a rectangle or the like in order to reduce the amount of calculation.

[0071] The selection range is determined on the XY plane based on the distance Z1 from the origin in the depth direction. Therefore, the selection unit 1120 can appropriately determine whether or not a virtual object is included in the selection range depending on the position of the virtual object in the depth direction. Therefore, the user can smoothly select multiple virtual objects arranged in a three-dimensional space as desired without causing any misalignment in appearance.

[0072] (Variation 1) A first modified example of the first embodiment will be described with reference to Fig. 3(E). In the first embodiment, a selection area 3100 is set based on a hand trajectory 3090. In contrast, in the first modified example, the selection area 3100 is specified based on position information of multiple points specified by the user's hand.

[0073] For example, the user can specify the selection area 3100 by pointing to two positions with the left hand 3020 and the right hand 3030. Specifically, the information processing device 1100 starts a virtual object selection mode when the distance between the user's left hand 3020 and right hand 3030 becomes equal to or less than a threshold, as shown in FIG.

[0074] The selection unit 1120 sets the position of the tip of the index finger of the left hand 3020 when the selection mode starts as the first vertex of the selection area 3100. Thereafter, the user moves the right hand 3030 as shown in FIG. 3(E) to specify the second vertex of the selection area 3100. The selection unit 1120 sets the position of the tip of the index finger when the movement of the right hand 3030 stops as the second vertex of the selection area 3100. The selection unit 1120 can set a rectangle as the selection area 3100, with the first vertex pointed to by the left hand 3020 and the second vertex pointed to by the right hand 3030 as vertices on a diagonal line.

[0075] The user may specify three or more points, not limited to two points, to specify the selection area 3100. For example, the selection area 3100 may be a circular area passing through three points specified by the user, or may be a rectangle passing through four points specified by the user. The user can easily specify the selection area 3100 by specifying the positions of multiple points.

[0076] (Variation 2) A second modified example of the first embodiment will be described with reference to Fig. 3(F). In the first embodiment, a selection area 3100 is specified based on a hand trajectory 3090. In contrast, in the second modified example, the selection area 3100 is specified by an area surrounded by a predetermined hand shape of a user. The predetermined hand shape is a shape formed by bringing two fingertips of one hand close to two fingertips of the other hand, or a shape formed by bringing two fingertips of one hand close to each other.

[0077] 3(F), the user forms a frame shape by making the thumb and index finger of each of the left hand 3200 and the right hand 3210 form a substantially right angle, and bringing the thumb of the left hand 3200 close to the index finger of the right hand 3210 and the index finger of the left hand 3200 close to the thumb of the right hand 3210. The selection unit 1120 can set the formed frame shape as the selection area 3100.

[0078] The predetermined hand shape may also be a shape formed by one hand. For example, the user forms a circle by connecting the tips of the thumb and index finger of one hand. The selection unit 1120 can set the formed circular area as the selection area 3100. The user can easily specify the selection area 3100 by forming the predetermined hand shape.

[0079] [Second embodiment] The first embodiment is an embodiment in which a selection range in a three-dimensional space is set based on a selection region 3100 specified by an operating object such as a user's hand. In contrast, the second embodiment is an embodiment in which a selection range in a three-dimensional space is set based on the trajectory of a laser beam-like object (hereinafter referred to as a ray) emitted from the position of the user's hand. The ray may be displayed as emanating from the hand by hand tracking, or may be displayed as emanating from a controller held in the user's hand.

[0080] In the second embodiment, the selection unit 1120 determines a range on a two-dimensional plane according to the depth distance from the ray emission position to the virtual object based on the emission angle of the ray with respect to the depth direction. , it is determined whether the virtual object is included in the selection range.

[0081] The configuration of the image processing system 100 according to the second embodiment is similar to that of the first embodiment. The following mainly describes the differences from the first embodiment. In the first embodiment, the selection unit 1120 sets the selection range based on a selection region 3100 specified by the user's hand, a controller, or the like. On the other hand, in the second embodiment, the selection unit 1120 sets the selection range based on a direction specified by a ray emitted from the user's hand or an XR controller held by the user, or the like.

[0082] The user can specify a selection range at a distant position by changing the direction of the ray. The selection unit 1120 can set the selection range, for example, by expanding a cone shape surrounded by the trajectory of the ray in the depth direction.

[0083] The ray is not limited to being displayed as a laser beam, and may be displayed as a point (pointer) that intersects with a virtual object or the like that exists in the direction in which the ray is emitted. In this case, the selection unit 1120 can set the selection range by expanding in the depth direction a cone-shaped shape surrounded by a line segment connecting the origin position from which the ray is emitted and the pointer. In the following description, the ray is an object displayed in the form of a laser beam, but this embodiment can also be applied to the case in which the ray is displayed as a pointer.

[0084] The selection unit 1120 can determine an area on the XY plane (area where the selection range intersects with the XY plane) at a distance Z1 in the depth direction (Z-axis direction) from the user's position based on the angle of the ray with respect to the depth direction.

[0085] 6 is a diagram for explaining the selection range according to the second embodiment. An origin 6010 is the position from which a ray is emitted, and corresponds to the position of a user's hand or fingertip, or the ray emission position of an XR controller held by the user. Unlike the first embodiment, the origin 6010 is not the position of the viewpoint from which the user observes the three-dimensional space, but the position from which the ray is emitted. The coordinates of the origin 6010 are (X0, Y0, Z0).

[0086] When the user specifies the selection range with a ray, the condition for starting the selection mode may be, for example, that the user changes the shape of his or her hand to a predetermined shape. The selection mode may be started by a user operation from an input device such as a button. The selection mode may also be started by a user's gaze or voice instruction. The selection unit 1120 sets the selection range based on the range enclosed by the trajectory of the ray from the start to the end of the selection mode.

[0087] The condition for ending the selection mode may be, as in the case of starting the selection mode, that the user changes the shape of his / her hand into a predetermined shape. The predetermined shape may be the same or different for starting and ending the selection mode. The selection mode may also be ended by a user operation from an input device, a user's gaze, or a voice instruction.

[0088] In the selection mode, the length of the ray may be, for example, the distance to the virtual object closest to the user, or the distance to the virtual object farthest from the user. The length of the ray may also be the average value of the distances to multiple virtual objects existing in the virtual space, or the distance to the object that the ray hits first after the selection mode starts. The length of the ray may be constant during the selection mode. By making the length of the ray constant, it becomes easier for the user to select a desired range.

[0089] When the angle of the selection range is specified by the trajectory of the ray, the selection unit 1120 selects the selected angle If the emission angle of the ray with respect to the front direction (Z-axis direction) is (θxs, θys), the coordinates of the selection range at a distance of Z1 are expressed as (X0+(Z1-Z0)×tanθxs, Y0+(Z1-Z0)×tanθys, Z1).

[0090] The selection unit 1120 can determine the area on the XY plane at Z coordinate Z1 of the selection range by repeating coordinate calculations using the angle between the ray moved by the user and the Z axis direction. In this way, the selection unit 1120 can determine the area where the selection range intersects with the XY plane according to the distance Z1 in the depth direction.

[0091] When the user selects a cone-shaped range by a ray, the selection unit 1120 may set the selected range by approximating the cone-shaped range selected by the user to a circumscribing square pyramid. By approximating the range selected by the user to a square pyramid, the selection unit 1120 can reduce the amount of calculation of coordinates.

[0092] In the second embodiment described above, the information processing device 1100 sets the selection range based on the trajectory of the ray. Specifically, the information processing device 1100 sets the selection range using the angle with respect to the depth direction specified by the user with the ray.

[0093] The selection range is determined on the XY plane based on the distance Z1 from the origin in the depth direction. Therefore, the selection unit 1120 can appropriately determine whether or not a virtual object is included in the selection range depending on the position of the virtual object in the depth direction. Therefore, the user can smoothly select multiple virtual objects arranged in a three-dimensional space as desired without causing any misalignment in appearance. Furthermore, by specifying the selection range with a ray, the user can select virtual objects with greater precision.

[0094] [Third embodiment] The first embodiment is an embodiment in which the selection range is set with the virtual viewpoint as the origin (origin 4010 in Figs. 4 and 5) regardless of the number of cameras included in the imaging unit 1010. In contrast, the third embodiment is an embodiment in which the imaging unit 1010 is a stereo camera having two cameras fixed to each other corresponding to the left and right eyes so as to be able to capture an image of the real space in the line of sight direction from the viewpoint position of the user.

[0095] In the third embodiment, the selection unit 1120 sets the selection range with the position of the user's dominant eye as the origin. If the imaging unit 1010 includes a camera disposed at approximately the same position as the user's dominant eye, the selection unit 1120 can set the position of the camera as the position of the origin.

[0096] The configuration of the image processing system 100 according to the third embodiment is the same as that of the first embodiment. The following mainly describes the differences from the first embodiment. The third embodiment is an embodiment in which a selection range is set with the position of the dominant eye as the origin, and the image processing system 100 has a configuration in which the user can set which of the left and right eyes is the dominant eye. The display device 1000 or the information processing device 1100 may display a menu screen for the user to set which eye is the dominant eye, and may accept a setting operation from the user. Furthermore, when there is no information on the dominant eye, the display device 1000 may automatically determine the dominant eye by a known technology and set the dominant eye of the user. Furthermore, the dominant eye of the user may be a preset eye.

[0097] Fig. 7 is a diagram for explaining a selection range according to the third embodiment. Using Fig. 7, the effect of using the position of the dominant eye as the origin will also be explained. In this embodiment, the camera corresponding to the dominant eye is assumed to be disposed at the position of the dominant eye.

[0098] An HMD 7040 worn by a user 7010 experiencing the MR space includes two cameras, a camera 7020 and a camera 7030. The cameras 7020 and 7030 are disposed at approximately the same positions as the user's left and right eyes. The user's dominant eye is the left eye, and the camera on the dominant eye side is the camera 7020. The origin of the observation direction is the center position of the camera 7020 on the dominant eye side. The coordinates of the origin are (X0, Y0, Z0).

[0099] The user specifies the selection region 3100 in the same manner as in the first embodiment. The selection unit 1120 can set the selection range by expanding the selection region 3100 in the depth direction using the origin and a position on the contour of the selection region 3100, in the same manner as in the case described in Fig. 4. Furthermore, the selection unit 1120 may set the selection range using the angle of view θ of the camera on the dominant eye side and an angle formed by a straight line connecting a position on the contour of the selection region 3100 and the origin with the depth direction, in the same manner as in the case described in Fig. 5.

[0100] The selection unit 1120 sets, as a selection range in three-dimensional space, a space enclosed by a plurality of straight lines (such as dotted lines 7050 and 7060 in FIG. 7) that extend from the origin (X0, Y0, Z0) through points on the contour of the selection area 3100 toward the field of view of the left eye. At a distance Z1 from the origin, the virtual object 3040 included within the selection range is in a selected state.

[0101] On the other hand, if the position of the camera 7030 on the non-dominant eye side is taken as the origin, the space enclosed by a plurality of straight lines (such as dashed lines 7070 and dashed lines 7080 in FIG. 7) that pass through the origin and the points on the contour of the selection area 3100 and extend in the direction of the field of view of the right eye is set as the selection range in the three-dimensional space. In this case, even if a user whose dominant eye is the left eye attempts to select virtual object 3040 at a distance of Z1, virtual object 3070 will be selected, and the intended range will not be selected. Therefore, when a stereo camera (twin-lens camera) is used as the imaging unit 1010, it is important to take the position of the camera on the dominant eye side as the origin.

[0102] If information about the dominant eye is not set, the origin of the observation direction (X0, Y0, Z0) may be the midpoint between the centers of the two cameras, rather than the center position of either the left or right camera. By setting the midpoint between the cameras as the origin, the error due to the parallax between the left and right eyes can be reduced by approximately half.

[0103] Furthermore, when the selection mode starts, the display unit 1020 of the HMD (display device 1000) may display an image captured by either the left or right camera on each of the left and right displays. For example, when the selection mode starts, the display unit 1020 switches the image displayed on the right eye display to an image captured by the left eye camera. Then, the selection unit 1120 sets the position of the camera corresponding to the image being displayed as the origin (X0, Y0, Z0) of the observation direction.

[0104] When the selection mode ends, the display unit 1020 returns the images displayed on the left and right displays of the HMD to the images captured by the corresponding cameras. For example, if the selection mode starts and the image displayed on the right eye display is switched to the image captured by the left eye camera, the display unit 1020 returns the image displayed on the right eye side to the image captured by the right eye camera.

[0105] When the selection mode starts, the same image captured by either the left or right camera is displayed on the left and right displays, which prevents the intended range from being selected due to misalignment between the left eye image and the right eye image. Therefore, the selection unit 1120 can set the selection range as intended by the user.

[0106] In the third embodiment described above, when the imaging unit 1010 is a twin-lens camera, the camera on the dominant eye side Since the selection range is set with the position of the dominant eye (the position of the dominant eye) as the origin, errors in the selection range are suppressed. Therefore, the user can smoothly select multiple virtual objects placed in three-dimensional space as desired, while suppressing the effects of parallax.

[0107] [Fourth embodiment] The fourth embodiment is an embodiment in which a selection range is set in the same manner as in the first to third embodiments, a virtual object is changed to a selected state, and then the selection range is designated to deselect a part of the selected virtual object. The selection unit 1120 can designate the selection range based on a predetermined operation by the user. The configuration of the image processing system 100 according to the fourth embodiment is the same as that of the first embodiment. The following mainly describes the differences from the first to third embodiments.

[0108] In the fourth embodiment, the selection unit 1120 specifies the depth direction of the selection range. First, in the selection mode, the selection unit 1120 changes a virtual object included in the selection range specified by the user to a selected state. Specifically, the selection unit 1120 determines an area of the XY plane included in the selection range based on the distance of the virtual object in the depth direction (Z-axis direction), and determines whether or not the virtual object is included in the selection range. If the virtual object is included in the selection range, the selection unit 1120 changes the virtual object to a selected state. When the virtual object is selected, the selection mode ends.

[0109] When the selection mode ends and a selection range designation mode for designating the depth direction of the selection range starts, the selection unit 1120 accepts a predetermined operation from the user for designating the depth direction of the selection range. In response to the predetermined operation by the user, the selection unit 1120 sets some of the selected virtual objects to a deselected state.

[0110] A predetermined operation for specifying the depth direction of the selection range is, for example, an operation of extending the index finger of the left hand in the depth direction, placing the tip of the index finger of the right hand on the index finger of the left hand, and moving it toward the back or front. When the tip of the index finger of the right hand is moved toward the back, the selection unit 1120 changes the selected virtual objects to a deselected state one by one, starting from the front. Conversely, when the tip of the index finger of the right hand is moved toward the front, the selection unit 1120 changes the selected virtual objects to a deselected state one by one, starting from the back.

[0111] 4, when the user specifies the selection region 3100, the three-dimensional selection range includes the virtual object 3040 and the virtual object 3070. The selection unit 1120 sets the virtual object 3040 and the virtual object 3070 to a selected state. When the selection range specification mode starts and the user places the tip of the index finger of the right hand on the index finger of the left hand and moves it toward the back, the selection unit 1120 changes the virtual object 3040 to a deselected state. The selection unit 1120 may set one virtual object to a deselected state when, for example, the finger of the right hand has moved a preset movement distance.

[0112] Note that the predetermined operation for specifying the depth direction of the selection range is not limited to the above operation. The predetermined operation may be an operation for moving the position of the user's hand in the depth direction. After the selection range is set and the selection mode is terminated, when the user moves one of his / her hands (e.g., the right hand) to the depth side in the Z axis direction, the selection unit 1120 may deselect the selected virtual objects one by one from the front side. Conversely, when the user moves one of his / her hands to the depth side in the Z axis direction, the selection unit 1120 may deselect the selected virtual objects one by one from the depth side.

[0113] The predetermined operation for specifying the depth direction of the selection range may be an operation for moving the thumb of one hand toward the back or toward the front in the Z-axis direction. The selection unit 1120 limits the depth direction of the selection range according to a change in the positional relationship between the tip of the thumb and the tip of the index finger of one hand. For example, the predetermined operation is an operation in which the user points the tip of the index finger toward the front and places the tip of the thumb on the index finger and moves it. When the user moves the thumb closer to the base of the index finger, the selection unit 1120 may limit the selection range to the back side.

[0114] The predetermined operation for specifying the depth direction of the selection range may be a pinch-in or pinch-out operation of hand gestures. For example, after the selection range is set and the selection mode is terminated, the user can limit or expand the depth direction of the selection range by moving the thumb and index finger of one hand closer to or farther apart.

[0115] When a pinch-in operation of bringing the thumb and index finger closer to each other is detected, the selection unit 1120 narrows both the front and back sides of the selection range. The selection unit 1120 deselects the virtual objects located on the front and back sides, and leaves the virtual object located in the middle in the selected state.

[0116] Furthermore, when a pinch-out operation of moving the thumb and index finger apart is detected, the selection unit 1120 may expand the selection range and set the deselected virtual object to a selected state again. Furthermore, after the depth direction of the selection range is specified by a pinch-in operation or a pinch-out operation, when the entire hand moves in the Z-axis direction while maintaining the distance between the thumb and index finger, the selection unit 1120 may shift the selection range in the Z-axis direction in accordance with the movement of the hand. Note that the selection unit 1120 may shift the selection range in the Z-axis direction not only when the entire hand of the user moves in the Z-axis direction, but also when the hand moves in the up-down direction (Y-axis direction) or the pinch-in / pinch-out direction.

[0117] In the above fourth embodiment, the information processing device 1100 limits the selection range in the depth direction based on a user operation. Therefore, even if a plurality of virtual objects arranged in the virtual space overlap in the depth direction, the user can smoothly select a desired virtual object as desired. In addition, the user is not limited to limiting the selection range in the depth direction with both hands, but can limit the selection range in the depth direction with an operation using one hand and easily select a desired virtual object.

[0118] [Fifth embodiment] In each of the above embodiments, each unit constituting the information processing device 1100 shown in Fig. 1 is configured with hardware. In a fifth embodiment, part of the configuration of the information processing device 1100 may be configured with software. The information processing device 1100 according to this embodiment is a computer that realizes part of the operations described in each of the above embodiments by executing software and implements the remaining operations (functions) as hardware.

[0119] 8 is a block diagram showing an example of the hardware configuration of a computer applicable to the information processing device 1100. A CPU 8001 uses programs and data stored in a RAM 8002 and a ROM 8003 to control the entire computer and execute each process of the information processing device 1100 described in each of the above embodiments.

[0120] The RAM 8002 has an area for temporarily storing programs and data loaded from an external storage device 8007 or a storage medium drive 8008. The RAM 8002 has an area for temporarily storing data received from an external device via an I / F (interface) 8009. The external device is, for example, the display device 1000. The data received from the external device is, for example, a real space image and an input value generated by an input device of the display device 1000 based on an operation from a user.

[0121] The RAM 8002 also has a work area that the CPU 8001 uses when executing various processes. That is, the RAM 8002 can provide various areas as appropriate. For example, the RAM 8002 also functions as the data storage unit 1150 shown in FIG.

[0122] The ROM 8003 is a non-volatile memory that stores the setting data and boot program of the computer.

[0123] The keyboard 8004 and the mouse 8005 are examples of operation input devices, and a computer user can input various instructions to the CPU 8001 by operating them.

[0124] The display unit 8006 is, for example, a CRT display or a liquid crystal display, and can display the results of processing by the CPU 8001 as images and characters. For example, the display unit 8006 can display messages for measuring the position and orientation of the display device 1000.

[0125] The external storage device 8007 is a large-capacity information storage device such as a hard disk drive, and stores an operating system (OS), programs for causing the CPU 8001 to execute various processes of the information processing device 1100, and data.

[0126] The programs stored in the external storage device 8007 include programs corresponding to the processes of the position and orientation acquisition unit 1110, the selection unit 1120, the image generation unit 1130, and the image synthesis unit 1140. The data stored in the external storage device 8007 includes the information described as known information and various setting information, in addition to the data of the virtual space. The programs and data saved in the external storage device 8007 are loaded into the RAM 8002 as appropriate under the control of the CPU 8001. The CPU 8001 executes each process of the information processing device 1100 by executing processes using the programs and data loaded into the RAM 8002. The external storage device 8007 may be used as the data storage unit 1150 shown in FIG. 1.

[0127] The storage medium drive 8008 reads out programs and data recorded on computer-readable storage media such as CD-ROMs or DVD-ROMs, and writes programs and data to these storage media. Note that some or all of the programs and data stored in the external storage device 8007 may be recorded on these storage media. The programs and data read by the storage medium drive 8008 from the storage media are output to the external storage device 8007 or the RAM 8002.

[0128] The I / F 8009 is an analog video port or a digital input / output port such as IEEE1394 for connecting the imaging unit 1010 of the display device 1000. The I / F 8009 may also be an Ethernet (registered trademark) port for outputting a composite image to the display unit 1020 of the display device 1000. Data received via the I / F 8009 is input to the RAM 8002 or the external storage device 8007. When a sensor system is used for the position and orientation acquisition unit 1110 to acquire position and orientation information, the I / F 8009 is used as an interface for connecting the sensor system. The bus 8010 connects the units illustrated in FIG. 8 to each other.

[0129] [Other embodiments] The present disclosure also includes a case where a software program is supplied to a system or device directly or remotely, and the computer of the system or device reads and executes the supplied program code to achieve the functions of each of the above embodiments. The program supplied to the system or device executes the process corresponding to the flowchart described in FIG. This is a program for

[0130] The functions in each of the above embodiments may be realized by a computer executing a read program, or may be realized in cooperation with an OS running on a computer based on instructions from the program. In this case, the functions in each of the embodiments are realized by the OS executing a part or all of the functions.

[0131] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0132] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A display control means for controlling the display of a virtual object so that the virtual object is placed in a three-dimensional space within a user's field of view; a selection means for selecting the virtual object included in a selection range in the three-dimensional space selected using an operating object at the position of the user's hand; 13. An information processing device comprising: (Configuration 2) The selection range is a three-dimensional range obtained by expanding in the depth direction a two-dimensional selection area designated by the user using the operating object. 2. The information processing device according to configuration 1. (Configuration 3) The information processing device described in configuration 2, characterized in that the selection range is a three-dimensional range obtained by extending the two-dimensional selection area in the depth direction using an origin corresponding to the user's position and a position on the contour of the two-dimensional selection area. (Configuration 4) The operating object is a user's hand. The two-dimensional selection area is an area enclosed by the locus of the user's hand or fingertip positions. 4. The information processing device according to configuration 2 or 3. (Configuration 5) The operating object is a user's hand. The two-dimensional selection area is an area specified based on position information of multiple points specified by the user's hands. 4. The information processing device according to configuration 2 or 3. (Configuration 6) The operating object is a user's hand. The two-dimensional selection area is an area enclosed by a predetermined hand shape of the user. 4. The information processing device according to configuration 2 or 3. (Configuration 7) The predetermined hand shape is a shape formed by bringing two fingertips of one hand close to two fingertips of the other hand, or a shape formed by bringing two fingertips of one hand close to each other. 7. The information processing device according to configuration 6. (Configuration 8) The selection range is set using an origin corresponding to the position of the user and a range of directions in which the operating object is pointed from the origin. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) 9. The information processing device according to any one of configurations 3 to 8, wherein the origin corresponding to the position of the user is the position of either the left or right eye of the user. (Configuration 10) The device further includes a setting unit for setting a dominant eye of the user, The origin corresponding to the position of the user is the position of the dominant eye of the user set by the setting means. 10. The information processing device according to configuration 9. (Configuration 11) The origin corresponding to the user's position is the midpoint between the user's left and right eyes. 9. The information processing device according to any one of configurations 3 to 8. (Configuration 12) an imaging unit configured to capture a first image for a right eye and a second image for a left eye; The display control means When an operation by the operating object for setting the selection range is started, the first image or the second image is displayed on a first display for a right eye and a second display for a left eye; When the virtual object is selected by the selection means, the display of the first display and the display of the second display are returned to the display of the first image and the second image, respectively. 12. The information processing device according to any one of configurations 1 to 11. (Configuration 13) The selection range is set based on the trajectory of a ray, which is a laser beam-like object emitted from the operating object. 2. The information processing device according to configuration 1. (Configuration 14) The selection means determines a range on a two-dimensional plane according to a depth distance from the emission position of the ray to the virtual object based on an emission angle of the ray with respect to a depth direction, and determines whether the virtual object is included in the selection range. 14. The information processing device according to configuration 13. (Configuration 15) The selection means specifies a range in the depth direction of the selection range based on a predetermined operation by the user. 15. The information processing device according to any one of configurations 1 to 14. (Configuration 16) the predetermined operation is an operation of moving a position of the user's hand in a depth direction, The selection means is When the user's hand moves in a direction approaching the user, the selection state is cancelled in order from the back of the virtual object; When the user's hand moves in a direction away from the user, the selection state is cancelled in order from the front virtual object. 16. The information processing device according to configuration 15. (Configuration 17) The display control means displays the selected virtual object in a manner different from that when the virtual object is not selected. 17. The information processing device according to any one of configurations 1 to 16. (Configuration 18) When the selection range is changed, the selection means deselects the virtual object that is no longer included in the selection range. 18. The information processing device according to any one of configurations 1 to 17. (Configuration 19) The selection means determines whether a center of gravity of the virtual object is included in the selection range, Whether or not the virtual object is included in the selection range is determined based on whether or not the entirety of the virtual object is included in the selection range, or whether or not a predetermined ratio or more of the virtual object is included in the selection range. 19. The information processing device according to any one of configurations 1 to 18. (Configuration 20) The operation object is a user's hand, The selection means acquires the position of the user's hand or fingertip from an image captured by an imaging unit, or acquires the position of the user's hand based on position and orientation information of a controller held in the user's hand. 20. The information processing device according to any one of configurations 1 to 19. (method) a display control step of controlling the display of a virtual object so that the virtual object is placed in a three-dimensional space that is within the user's field of view; a selection step of selecting the virtual object included in a selection range in the three-dimensional space selected using an operating object at the position of the user's hand; An information processing method comprising causing a computer to execute the steps of: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 20. [Explanation of symbols]

[0133] 1100: information processing device, 1130: image generating unit, 1120: selection unit, 8001: CPU

Claims

1. a display control means for controlling the display of a virtual object so that the virtual object is positioned in a three-dimensional space within the user's field of view; a selection means for selecting the virtual object included in a selection range in the three-dimensional space obtained by expanding a two-dimensional selection area designated using an operating object at the position of the user's hand in a depth direction; and The selection range is a three-dimensional range extending in a frustum shape bounded by a plurality of straight lines extending from an origin corresponding to the user's position through a point on the outline of the two-dimensional selection area in the direction of the user's field of view.

1. An information processing device comprising:

2. The operation object is the user's hand. The two-dimensional selection area is an area surrounded by the locus of the user's hand or fingertip positions.