Information processing apparatus and information processing method

Abstract

To provide an information processing apparatus configured to allow a user to easily select a desired virtual object from among a plurality of virtual objects superposed in a depth direction in a three-dimensional space.SOLUTION: An information processing apparatus includes: display control means which performs display control of a virtual object so as to be arranged in a three-dimensional space which is a field of view of a user; setting means which sets a selection range in the three-dimensional space on the basis of a plurality of positions located at different distances from the user, the positions being designated by an operator located in a position of a hand of the user; and selection means which switches the state of the virtual object between a selected state and a not-selected state on the basis of the selection range.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, Virtual Reality (VR) systems and Mixed Reality (MR) systems have been developed with the aim of seamlessly connecting real and virtual spaces. These systems are used with head mounted displays (HMDs), which are glasses-type devices equipped with small displays.

[0003] A method has been proposed for recognizing the user's hand movements and using them for operation as a user interface (UI) when wearing an HMD. For example, Patent Document 1 describes a method for arranging video content on an extension of an area surrounded by the user's fingers. Patent Document 2 describes a method for changing the selected object from among multiple overlapping objects when a specific gesture is received from the user interface. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-100032 A [Patent Document 2] Patent Publication No. 2021-197024 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the operation of selecting a desired object from a plurality of overlapping objects (virtual objects) can be cumbersome.

[0006] The present invention aims to provide an information processing device that enables a user to easily select a desired virtual object from a plurality of virtual objects that are arranged overlapping each other in the depth direction in a three-dimensional space. [Means for solving the problem]

[0007] The information processing device according to the present invention is characterized by having a display control means for controlling the display of a virtual object so that it is positioned within a three-dimensional space within the user's field of vision, a setting means for setting a selection range in the three-dimensional space based on a plurality of positions at different distances from the user that are specified using an operating object located at the position of the user's hand, and a selection means for switching the state of the virtual object between a selected state and a deselected state based on the selection range. Effect of the Invention

[0008] According to the present invention, a user can easily select a desired virtual object from a plurality of virtual objects arranged overlapping each other in the depth direction in a three-dimensional space. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram illustrating an example of a functional configuration of the image processing system. [Diagram 2] 4 is a flowchart showing a process of the information processing device according to the first embodiment. [Diagram 3] 4A to 4C are diagrams illustrating an example of an operation for selecting a virtual object in the first embodiment. [Figure 4] 13A to 13C are diagrams illustrating an example of an operation for selecting a virtual object in the second embodiment. [Diagram 5] 13A to 13C are diagrams illustrating a method for setting a selection range in the second embodiment. [Figure 6] 13A to 13C are diagrams illustrating an example of an operation for selecting a virtual object in the third embodiment. [Figure 7] FIG. 11 is an elevation view illustrating the positional relationship between a user's hand and a virtual object. [Figure 8]FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments for carrying out the present invention will be described in detail with reference to the drawings. Note that the embodiments described below are examples of implementation means, and can be appropriately modified depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments. In addition, it is also possible to appropriately combine parts of each of the following embodiments and each of the modified examples without departing from the gist of the present invention.

[0011] [First embodiment] 1 is a block diagram showing an example of a functional configuration of an image processing system 100 according to this embodiment. The image processing system 100 is a system for presenting a mixed reality space (MR space) that combines a real space and a virtual space to a person (user) experiencing the system. The image processing system 100 performs display control so that a virtual object is placed in a three-dimensional space that is within the user's field of vision. The user can select a desired virtual object by specifying (selecting) a selection range in the three-dimensional space.

[0012] In this embodiment, it is assumed that an MR space is presented to a user by displaying a composite image obtained by combining an image of real space with an image of virtual space rendered by computer graphics (CG).

[0013] The image processing system 100 includes a display device 1000 and an information processing device 1100. The information processing device 1100 can synthesize an image of real space captured from the display device 1000 with an image of virtual space generated by the information processing device 1100, and output the synthesized image to the display device 1000 as a mixed reality image (MR image).

[0014] The image processing system 100 relates to a system that displays an image of a virtual space, and is not limited to an MR (mixed reality) system that displays an MR image that combines an image of a real space with an image of a virtual space. The image processing system 100 can also be applied to a VR (virtual reality) system that presents only an image of a virtual space to a user, or an AR (augmented reality) system that presents an image of a virtual space to a user by transmitting the real space.

[0015] The display device 1000 includes an imaging unit 1010. The imaging unit 1010 captures images of real space continuously in time series, and outputs the captured images of real space to the information processing device 1100. The imaging unit 1010 may include a stereo camera consisting of two cameras fixed to each other so as to be able to capture images of real space in the line of sight direction from the viewpoint position of the user.

[0016] The display device 1000 includes a display unit 1020. The display unit 1020 displays the MR image output from the information processing device 1100. The display unit 1020 may include two displays arranged corresponding to the left and right eyes of the user, respectively. In this case, the left-eye display corresponding to the left eye of the user displays the MR image for the left eye, and the right-eye display corresponding to the right eye of the user displays the MR image for the right eye.

[0017] The display device 1000 is, for example, a head-mounted display device (HMD) that the user wears on the head. However, the display device 1000 is not limited to an HMD, and may be a handheld display (HHD). An HHD is a handheld display that the user holds in his or her hand. The display device 1000 may be a display terminal such as a tablet or a smartphone.

[0018] The information processing device 1100 and the display device 1000 are connected to each other so as to be able to perform data communication with each other. The connection between the information processing device 1100 and the display device 1000 may be wired or wireless. Furthermore, the information processing device 1100 may be disposed inside the housing of the display device 1000.

[0019] The information processing device 1100 includes a position and orientation acquisition unit 1110 , a selection unit 1120 , an image generation unit 1130 , an image synthesis unit 1140 , and a data storage unit 1150 .

[0020] The position and orientation acquisition unit 1110 acquires the position and orientation of the imaging unit 1010 in the world coordinate system and the position of the observer's (user's) hand. Specifically, the position and orientation acquisition unit 1110 extracts markers assigned to the world coordinate system from an image of real space captured by the imaging unit 1010. The position and orientation acquisition unit 1110 acquires the position and orientation of the imaging unit 1010 in the world coordinate system based on the position and orientation of the markers, and outputs the acquired position and orientation information of the imaging unit 1010 to the data storage unit 1150.

[0021] The position and orientation acquisition unit 1110 extracts a feature region of the hand from an image of real space captured by the imaging unit 1010. The position and orientation acquisition unit 1110 acquires position information of each part of the hand using the extracted feature region of the hand and hand shape information stored in the data storage unit 1150. The position information of each part of the hand may be position information of a part of the hand such as the fingertips or joints of each finger. The position and orientation acquisition unit 1110 only needs to acquire position information of parts used by the selection unit 1120 to set a selection range. For example, when the user specifies a selection range with a fingertip, it is sufficient to acquire position information of the fingertip.

[0022] Note that the method of acquiring the position and orientation of the image capturing unit 1010 is not limited to the above method. For example, the position and orientation acquiring unit 1110 may perform Simultaneous Localization and Mapping (SLAM) processing based on feature points captured in an image. The position and orientation acquiring unit 1110 can obtain the position and orientation of the image capturing unit 1010 and the position and orientation in an individual coordinate system by SLAM processing.

[0023] Furthermore, the position and orientation of the imaging unit 1010 may be obtained by attaching a sensor whose relative position and orientation with respect to the imaging unit 1010 is known to the display device 1000 and using a measurement value from the sensor. The position and orientation acquisition unit 1110 can obtain the position and orientation of the imaging unit 1010 in the world coordinate system by converting the measurement value from the sensor based on the position and orientation of the sensor relative to the imaging unit 1010. Furthermore, the position and orientation acquisition unit 1110 may obtain the position and orientation of the imaging unit 1010 by using a motion capture system.

[0024] Also, the method of acquiring the position of the user's hand is not limited to the above method. For example, when the imaging unit 1010 is a monocular camera, the position and orientation acquisition unit 1110 can acquire the position of the hand by a ToF (Time of flight) sensor. Also, when the imaging unit 1010 has multiple cameras with different positions and orientations, the position and orientation acquisition unit 1110 may acquire the position of the hand based on images captured by the multiple cameras. The position and orientation acquisition unit 1110 can obtain the depth of the entire image from the stereo image by a method such as SemiGlobalMatching (SGM), and acquire the position of the hand using the depth information and hand shape information.

[0025] The position and orientation acquisition unit 1110 may acquire the hand position by having the user wear gloves equipped with sensors capable of acquiring the positions of the joints of the hand. The position of the hand may be acquired using a motion capture system. Also, when the user performs an operation using an MR controller placed at the position of the hand, the position and orientation acquisition unit 1110 may acquire the position of the controller as the position of the hand.

[0026] The selection unit 1120 sets a selection range in the virtual space based on the position of the user's hand acquired by the position and orientation acquisition unit 1110. The selection unit 1120 changes the selection state of the virtual space data (virtual object) included in the selection range.

[0027] The image generating unit 1130 constructs a virtual space based on the virtual space data stored in the data storage unit 1150. The virtual space data includes data related to each virtual object constituting the virtual space, data related to a real-world approximation object obtained by importing three-dimensional shape information of a real object acquired from a real space into the virtual space, and data related to a light source that illuminates the virtual space.

[0028] The image generating unit 1130 sets a virtual viewpoint based on the position and orientation of the imaging unit 1010 acquired by the position and orientation acquiring unit 1110. The image generating unit 1130 can set, for example, the position of the user's dominant eye or the midpoint between the left and right eyes as the virtual viewpoint. The user's dominant eye can be set in advance. When the imaging unit 1010 has multiple cameras, the image generating unit 1130 can set a virtual viewpoint based on the positional relationship between the positions of the cameras when the user is wearing the HMD (display device 1000) and the positions of the user's eyes.

[0029] The image generating unit 1130 generates an image of the virtual space seen from the set viewpoint (virtual space image). Note that since the technology for generating an image of the virtual space seen from a viewpoint having a predetermined position and orientation is well known, a detailed description thereof will be omitted.

[0030] The image synthesis unit 1140 synthesizes the image of the virtual space generated by the image generation unit 1130 and the image of the real space captured by the imaging unit 1010 to generate an MR image, which is an image of a three-dimensional space. The image synthesis unit 1140 outputs the generated MR image to the display unit 1020.

[0031] The data storage unit 1150 includes a RAM, a hard disk drive device, etc., and stores the various pieces of information described above. The data storage unit 1150 also stores information that will be described as known information and various types of setting information.

[0032] FIG. 2 is a flowchart showing an example of a process in which the information processing device 1100 generates an MR image and outputs it to the display device 1000.

[0033] In step S2010, the position and orientation acquisition unit 1110 acquires the position and orientation of the imaging unit 1010 and the position of the user's hand. The position of the user's hand includes information such as the positions of each part of the hand (e.g., fingertips, joints, etc.) and the position of a controller worn or held on the hand. Of this information, the position and orientation acquisition unit 1110 only needs to acquire information used to set a selection range.

[0034] In step S2020, the selection unit 1120 determines whether or not the mode is a mode for selecting a virtual object (selection mode). A method for determining whether or not the mode has shifted to the selection mode will be described later with reference to FIG. 3. If it is determined that the mode is the selection mode, the process proceeds to step S2030. If it is determined that the mode is not the selection mode, the process proceeds to step S2050.

[0035] In step S2030, the selection unit 1120 updates a selection range for changing the selection state of a virtual object that is arranged and displayed in a virtual space, based on the position of the user's hand. The selection range is a range in a three-dimensional space that is specified using an operating object at the position of the user's hand. The operating object may be the user's hand or a controller. In the following description, the operating object is the user's hand.

[0036] In step S2040, the selection unit 1120 updates the selection state of the virtual objects included in the selection range based on the updated selection range. The selection unit 1120 switches the state of the virtual objects included in the selection range to a selected state, and switches the state of the virtual objects not included in the selection range to a deselected state. Note that the virtual objects included in the selection range may be virtual objects whose entirety is included in the selection range, or may be virtual objects whose part (a predetermined percentage or more) is included. Furthermore, the virtual objects included in the selection range may be virtual objects whose center of gravity is included in the selection range.

[0037] In step S2050, the image generating unit 1130 generates an image of the virtual space seen from the virtual viewpoint, using the information on the position and orientation of the imaging unit 1010 acquired in step S2010.

[0038] In step S2060, the image synthesis unit 1140 synthesizes the image of the virtual space generated in step S2050 with the image of the real space captured by the imaging unit 1010 (real space image) to generate an MR image.

[0039] In step S2070, the information processing device 1100 determines whether or not the termination condition is satisfied. For example, the information processing device 1100 can determine that the termination condition is satisfied when an instruction to terminate the process of generating an MR image is input. If the termination condition is satisfied, the information processing device 1100 terminates the process shown in Fig. 2. If the termination condition is not satisfied, the information processing device 1100 returns to the process of step S2010.

[0040] Figure 3 is a diagram for explaining an example of an operation for selecting a virtual object in the first embodiment. Figures 3(A) to 3(E) show examples of a composite image (MR image) generated by the image composition unit 1140. Figures 3(A) to 3(E) also show an example in which the selection unit 1120 sets a selection range in a virtual space and changes the selection state of a virtual object.

[0041] The real space image 3010 is an image representing the real space, and is an image captured by the imaging unit 1010. The real space image 3010 is synthesized with an image of a virtual space in which various objects are arranged.

[0042] In a composite image obtained by combining a real space image 3010 and a virtual space image, objects representing the user's left hand 3020 and right hand 3030 are arranged. In addition, objects representing virtual objects 3040, 3050, 3060, and 3070 are arranged in the composite image. Coordinate axes 3080 indicate a coordinate system in the virtual space. The X-axis direction corresponds to the horizontal direction of the MR image displayed on the display. The Y-axis direction corresponds to the vertical direction of the MR image displayed on the display. The Z-axis direction corresponds to the depth direction of the virtual space.

[0043] 3A shows a state where the selection state of a virtual object in a virtual space has not been changed. The image generating unit 1130 generates a virtual space image based on the position and orientation of the imaging unit 1010 acquired by the position and orientation acquiring unit 1110. The image synthesizing unit 1140 synthesizes the virtual space image with the real space image captured by the imaging unit 1010 to generate an MR image (synthetic image).

[0044] Fig. 3(B) shows a state in which a selection mode for changing the selection state of a virtual object starts. In the example of Fig. 3(B), the selection mode starts when the distance between the user's left hand 3020 and right hand 3030 becomes equal to or smaller than a threshold.

[0045] The position and orientation acquisition unit 1110 acquires the positions of the user's left hand 3020 and right hand 3030. In the example of FIG. 3B, the position and orientation acquisition unit 1110 acquires the positions of the left and right index fingers as the positions of the hands. The selection unit 1120 sets a selection range based on the hand position information, and changes the selection state of a virtual object included in the selection range.

[0046] The condition for starting the selection mode may be that the amount of hand movement in a predetermined time period is equal to or less than a threshold, or that the distance between the left hand and the right hand is equal to or less than a threshold for a predetermined time period. The condition for starting the selection mode may be that the user changes the shape of the hand to a predetermined shape. The selection mode may be started by a user operation from an input device such as a button. The selection mode may also be started by a user's gaze or voice instruction.

[0047] Fig. 3(C) shows a state in which the user's left hand 3020 and right hand 3030 move from the state of Fig. 3(B) to surround the virtual object, and a selection range 3090 is set. In the example of Fig. 3(C), the left hand 3020 moves forward and the right hand moves backward, moving to positions at different distances from the user. The selection unit 1120 sets the selection range 3090 using three-axis information of the coordinate axes 3080, with the position of the tip of the index finger of the left hand 3020 and the position of the tip of the index finger of the right hand 3030 as a diagonal line. The set selection range 3090 may be indicated by a dotted line as shown in Fig. 3(C).

[0048] The selection range 3090 is preferably in a frustum shape with the user side as the upper surface. For example, the frustum shape is expanded in the direction in which the virtual object is placed, passing through a plurality of positions specified by the user's hand (position of the user's hand specifying the selection range) from the reference position, with the user's eye position (viewpoint), as the reference position. By expanding the selection range 3090 toward the back, the discrepancy between the set selection range 3090 and the range visible to the user is reduced. Note that the user's eye position may be the position of the dominant eye.

[0049] Since the selection range 3090 includes the virtual object 3070, the selection unit 1120 changes the virtual object 3070 to a selected state. The virtual object 3070 in the selected state may be displayed in an emphasized manner, such as by making the outline thicker than other virtual objects or by highlighting the virtual object. Note that the method of displaying a virtual object in a selected state may be any display mode different from that in a non-selected state (deselected state), and may be a wireframe display, or may be displayed in a color or transparency different from that in the deselected state.

[0050] Whether or not a virtual object is included in the selection range is not limited to being determined based on whether a part of the virtual object (a predetermined ratio or more) is included in the selection range. Whether or not a virtual object is included in the selection range may be determined based on whether the entire virtual object (each vertex of a rectangular parallelepiped circumscribing the virtual object) is included in the selection range or whether the center of gravity of the virtual object is included in the selection range.

[0051] Fig. 3(D) shows a state in which the user's left hand 3020 has further moved in the X-axis direction and right hand 3030 has further moved in the Z-axis direction from the state in Fig. 3(C). Selection range 3100 in the state of Fig. 3(D) is larger than selection range 3090 in Fig. 3(C) and includes virtual object 3070 and virtual object 3040. The selection unit 1120 selects virtual object 3040 in addition to virtual object 3070.

[0052] Fig. 3(E) shows a state in which the amount of movement of the user's left hand 3020 and right hand 3030 in a predetermined time period from the state in Fig. 3(D) becomes equal to or less than a threshold. When the selection range 3100 is set and the selection state of the virtual object is changed, the selection mode ends. When the selection mode ends, the selection range 3100 shown by the dotted line may be hidden.

[0053] The condition for terminating the selection mode is that the movement of the hand for a predetermined period of time is equal to or less than a threshold value, but is not limited to this. The condition for terminating the selection mode may be that the shape of the hand is changed, such as bending the fingertips. The selection mode may also be terminated by a user operation from an input device such as a button, The game may be ended by the user's gaze or voice command.

[0054] In the above first embodiment, the selection unit 1120 sets a selection range based on a plurality of positions at different distances from the user, which are specified by the fingertips of the user's hand. Since the depth direction range of the selection range is specified based on a plurality of positions at different distances from the user, the user can easily select a desired virtual object from a plurality of virtual objects that are arranged overlapping each other in the depth direction in a three-dimensional space.

[0055] [Variations] The first embodiment described above shows an example in which the user specifies the selection range using both hands. In a modified example, the user may specify a selection range by specifying a plurality of positions at different distances from the user with one hand. For example, the user can specify the selection range by sequentially confirming the position pointed to by the index finger of the left hand 3020 and the position pointed to by the index finger of the right hand 3030 in FIG. 3(C) with one hand. For example, when the amount of movement of one hand during a predetermined period of time becomes equal to or less than a threshold, the selection unit 1120 can confirm the position pointed to by the one hand as the position specified by the user. The user can select a desired virtual object with one hand even when the other hand is holding an HHD or a tablet, etc.

[0056] [Second embodiment] The first embodiment is an embodiment in which the selection range is set using information on the position of the fingertip, whereas the second embodiment is an embodiment in which the selection range is set using information on the positions of multiple parts of the hand.

[0057] Fig. 4 is a diagram for explaining an example of an operation for selecting a virtual object in the second embodiment. Fig. 4(A) to 4(E) show examples of a composite image (MR image) generated by the image composition unit 1140. The same objects as in Fig. 3 are given the same numbers and their description will be omitted. Fig. 4(A) shows the same state as Fig. 3(A).

[0058] Fig. 4(B) shows a state in which a selection mode for changing the selection state of a virtual object starts, similarly to Fig. 3(B). The position and orientation acquisition unit 1110 acquires the positions of the user's left hand 4010 and right hand 4020, and detects that the thumbs and index fingers of the left hand 4010 and right hand 4020 are each in an L-shape to form a rectangular frame shape. When the frame shape formed by the user's hand is detected, the selection unit 1120 can start the selection mode.

[0059] The left hand 4010 and the right hand 4020 are not limited to forming a rectangle with their fingers in contact with each other, but may form a rectangle with their fingers separated from each other. In addition, in FIG. 4B, an example is shown in which the frame shape is formed in front of the user so as to be parallel to the XY plane, but the frame shape may be formed at a position shifted from the front of the user, or may be formed with an inclination with respect to the XY plane. In addition, the frame shape may be formed with each side inclined with respect to the X-axis and the Y-axis.

[0060] FIG. 4(C) shows a state in which the user's left hand 4010 and right hand 4020 have moved apart from the state shown in FIG. 4(B), and the selection unit 1120 has set a selection range 4030 based on the shape of the moved hands. The selection unit 1120 sets a selection range 4030 based on position information of the thumb and index finger of the user's left hand 4010 and position information of the thumb and index finger of the user's right hand 4020. The set selection range 4030 may be indicated by a dotted line as shown in FIG. 4(C). Details of a method for setting the selection range 4030 using position information of the user's thumb and index finger will be described later with reference to FIG. 5.

[0061] Fig. 4(D) shows a state in which the user's left hand 4010 has moved away from right hand 4020 from the state shown in Fig. 4(C), and selection range 4030 has expanded to selection range 4040. Selection range 4040 includes virtual object 3040 and virtual object 3070, and virtual objects 3040 and 3070 are displayed in a manner indicating that they are in a selected state.

[0062] Fig. 4(E) shows a state in which the amount of movement of the user's left hand 4010 and right hand 4020 in a predetermined time period from the state in Fig. 4(D) becomes equal to or less than a threshold. When the selection range 4040 is set and the selection state of the virtual object is changed, the selection mode ends. When the selection mode ends, the selection range 4040 shown by the dotted line may be hidden.

[0063] Fig. 5 is a diagram for explaining a method for setting a selection range in the second embodiment, and shows the procedure in which the selection unit 1120 sets the selection range 4030 in Fig. 4(C). The same objects as in Fig. 4 are given the same numbers, and their explanations are omitted.

[0064] 5(A) shows a plane 4030a passing through the thumb and index finger of a user's left hand 4010, and a plane 4030b passing through the thumb and index finger of a user's right hand 4020. The planes 4030a and 4030b are planes that include two opposing sides of the selection range 4030.

[0065] 5(B) is an elevational view of the user 5010 in FIG. 5(A), the positional relationship between the user's left hand 4010 and right hand 4020, and the surfaces 4030a and 4030b, as viewed from the side of the user 5010. The left hand 4010 and right hand 4020 of the user 5010 are at different positions in the depth direction (Z-axis direction). The selection unit 1120 defines the surface 4030a based on position information of the thumb and index finger of the user's left hand 4010. The selection unit 1120 also defines the surface 4030b based on position information of the thumb and index finger of the user's right hand 4020.

[0066] Fig. 5(C) shows a state in which axis 4030c extends from the position of left hand 4010 on surface 4030a to surface 4030b, and axis 4030d extends from the position of right hand 4020 on surface 4030b to surface 4030a. Fig. 5(D) is a diagram showing the positional relationship between user 5010, user's left hand 4010 and right hand 4020, axis 4030c, and axis 4030d in Fig. 5(C) as viewed from the side of user 5010.

[0067] Axis 4030c may extend from the position of left hand 4010 (the position of the base of the thumb and index finger) to surface 4030b so as to be perpendicular to surface 4030a or surface 4030b. Axis 4030d may extend from the position of right hand 4020 (the position of the base of the thumb and index finger) to surface 4030a so as to be perpendicular to surface 4030a or surface 4030b. Furthermore, axes 4030c and 4030d may be determined so that selection range 4030 has a frustum shape with the back side as the base.

[0068] Fig. 5(E) shows a state in which each face of the selection range 4030 is defined using an intersection 4030e between the axis 4030c and the face 4030b, and an intersection 4030f between the axis 4030d and the face 4030a. Fig. 5(F) is an elevational view of the user 5010 in Fig. 5(E) as viewed from the side of the user 5010, showing the positional relationship between the user's left hand 4010 and right hand 4020, and the intersections 4030e and 4030f.

[0069] For example, the selection unit 1120 sets a rectangular area 4030B on a surface 4030b whose diagonal line connects intersection point 4030e and the position of right hand 4020. The selection unit 1120 also sets a rectangular area 4030A on a surface 4030a whose diagonal line connects intersection point 4030f and the position of left hand 4010. The selection unit 1120 can determine the selection range 4030 by considering rectangular area 4030A and rectangular area 4030B as two opposing surfaces.

[0070] FIG. 5(G) shows a state in which the selection range 4030 shown in FIG. 4(C) has been set. FIG. 5(H) is an elevational view of the positional relationship between the user 5010 in FIG. 5(G) and the set selection range 4030, as seen from the side of the user 5010. The selection unit 1120 may adjust the sizes of the rectangular region 4030A and the rectangular region 4030B to make the selection range 4030 a frustum shape with the back side being the bottom. Making the selection range 4030 a frustum shape reduces the deviation between the selection range 4030 and the range visible to the user. Therefore, the user can easily select a desired virtual object.

[0071] In the above second embodiment, the selection unit 1120 sets a selection range based on a plurality of positions at different distances from the user, which are specified using a plurality of parts of the user's hand. Since the depth direction range of the selection range is specified based on a plurality of positions at different distances from the user, the user can easily select a desired virtual object from a plurality of virtual objects that are arranged overlapping each other in the depth direction in the three-dimensional space.

[0072] [Variations] In the above-described second embodiment, in FIG. 4(B), an example is shown in which the shape of the user's hand is a rectangle formed by the thumbs and index fingers of both hands, but the shape of the user's hand may be a circle or a part of a circle formed by the thumbs and index fingers of both hands making an arc. In this case, the selection unit 1120 sets a selection range in the shape of a truncated cone. The user can set a selection range in the shape of a truncated pyramid or a truncated cone by changing the shape of the fingers according to the virtual object to be selected.

[0073] In the second embodiment, the selection unit 1120 sets the selection range using the position information of the thumb and index finger as the multiple pieces of position information of the user's hand, but the selection unit 1120 may use information on the three-dimensional outline of the hand. By using the information on the three-dimensional outline of the hand, the selection unit 1120 can also set the selection range along the outline of the hand, such as an area surrounded by the user's entire hands.

[0074] [Third embodiment] In the first and second embodiments, it is assumed that the virtual objects are placed within the reach of the user, whereas the third embodiment is an embodiment in which virtual objects are placed also within the reach of the user.

[0075] In the third embodiment, the selection unit 1120 sets the selection range based on the relative positional relationship between the hand as the operating body and the virtual object. The selection unit 1120 associates the virtual object with the range in the depth direction in which the operating body (hand) moves, and sets the selection range so as to include the virtual object corresponding to the range in the depth direction of multiple positions specified by the operating body.

[0076] Specifically, for example, the selection unit 1120 sets the selection range using the range in the depth direction in which the hand moves and the number of virtual objects present in the range obtained by expanding the multiple positions (frame shape, etc.) designated by the user's hand in the depth direction. The selection unit 1120 divides the range in the depth direction in which the hand moves by a division line equal to the number of virtual objects, and can set the selection range based on the positional relationship between the user's hand that designates the selection range and the division line. Note that in the following description, it is assumed that the range in the depth direction in which the hand moves is viewed in a plane, and the range in which the hand moves is described as being divided by a division line, but in reality, it is divided by a boundary such as a surface or area. The range in which the hand moves may be divided so that a virtual object can be selected based on multiple positions designated by the user's operating body, and may be divided by a boundary (such as a surface or area) based on the number and position of the virtual objects.

[0077] Fig. 6 is a diagram for explaining an example of an operation for selecting a virtual object in the third embodiment. Fig. 6(A) to 6(E) show examples of a composite image (MR image) generated by the image composition unit 1140. The same objects as those in Fig. 3 are given the same numbers, and their description will be omitted.

[0078] In the composite image obtained by combining the real space image 3010 and the virtual space image, objects representing the user's left hand 6050 and right hand 6060 are placed. In the composite image, objects representing virtual objects 6010, 6020, 6030, and 6040 are placed in positions that the user cannot reach. Fig. 7 is an elevational view showing the positional relationship in the depth direction between the user viewing the image shown in Fig. 6 and the virtual objects. In Fig. 7, the same objects as in Fig. 6 are denoted by the same numbers, and their explanations are omitted.

[0079] FIG. 7 shows a schematic diagram of a user 7010, boundaries 7080 and 7090 of the user's 7010 field of view, and boundaries 7100, 7110, 7120 connecting the user's 7010 viewpoint and hand position.

[0080] Position 7020 in the depth direction indicates the closest distance at which the camera of the imaging unit 1010 can recognize the position of the user's hand, or the position of the near plane (the nearest plane) of the viewing frustum, which is the range captured by the camera. The closest distance may be a preset value, or a value adjusted according to the display device 1000. Position 7070 indicates the farthest position at which the user can reach out their hand. The farthest distance may be a preset value, or a value adjusted for each user may be set.

[0081] A range from position 7020 to position 7070, which is a range in the depth direction in which the hand moves (hereinafter, movement range W), is divided by division lines 7030, 7040, 7050, and 7060. The number of division lines dividing the hand movement range W is determined according to the number of virtual objects arranged in the range surrounded by the boundary line 7100 and the boundary line 7110 connecting the viewpoint of the user 7010 and the position of the hand. In the example of FIG. 7, the number of virtual objects in the range surrounded by the boundary line 7100 and the boundary line 7110 is four. Therefore, the selection unit 1120 divides the hand movement range W at the positions of the four division lines 7030, 7040, 7050, and 7060. Note that the image generation unit 1130 may generate objects indicating boundaries (surfaces or regions) corresponding to these division lines, arrange them in the virtual space, and display them on the display unit 1020. Furthermore, the image generating unit 1130 may not generate an object corresponding to the division line so that the object is not displayed on the display unit 1020.

[0082] Each division line may divide the hand movement range W at equal intervals as illustrated in FIG. 7. Also, each division line may divide the hand movement range W at intervals based on the interval between the corresponding virtual objects. For example, the distance between the virtual object 6030 and the virtual object 6020 is longer than the distance between the virtual object 6020 and the virtual object 6010. In this case, the selection unit 1120 determines the position of the division line so that the distance between the division line 7040 and the division line 7050 is longer than the distance between the division line 7050 and the division line 7060. In this way, the selection unit 1120 can associate the virtual objects with the hand movement range W at equal intervals or at intervals based on the interval between the virtual objects. By dividing the hand movement range W according to the distance between the virtual objects, the user can specify the range in the depth direction of the selection range without feeling uncomfortable according to the distance (interval) between the actual virtual objects.

[0083] However, if the positions of the multiple virtual objects in the depth direction are approximately equal, it is difficult to select an intended virtual object from the virtual objects in the approximately equal positions if the position of the division line is determined according to the distance between the virtual objects. Therefore, when controlling the display of multiple virtual objects, the selection unit 1120 may associate the multiple virtual objects with the range in the depth direction in which the operating body moves at intervals of a predetermined threshold or more. The predetermined threshold is set so that the positions of the multiple virtual objects in the depth direction are approximately equal to each other. It is sufficient that the distance between adjacent dividing lines is set to a value that allows the user to select the selection state of equal virtual objects as intended. By setting the distance between adjacent dividing lines to a predetermined threshold or more, the user can set the selection range so that the desired virtual object is included, even if the positions of multiple virtual objects in the depth direction are approximately equal.

[0084] When a division line 7030 is set within a range in the depth direction specified by the user with his / her hand, the selection unit 1120 sets the selection range so as to include a virtual object 6040 corresponding to the division line 7030. Similarly, when division lines 7040, 7050, and 7060 are set within a range in the depth direction specified by the user with his / her hand, the selection unit 1120 sets the selection range so as to include virtual objects 6030, 6020, and 6010, respectively.

[0085] 6(A) shows a state where the selection state of a virtual object in a virtual space has not been changed. The image generating unit 1130 generates a virtual space image based on the position and orientation of the imaging unit 1010 acquired by the position and orientation acquiring unit 1110. The image synthesizing unit 1140 synthesizes the virtual space image with the real space image captured by the imaging unit 1010 to generate an MR image (synthetic image).

[0086] Fig. 6(B) shows a state where a selection mode for changing the selection state of a virtual object starts. In Fig. 6(B), the distance from the user 7010 to the user's left hand 6050 and right hand 6060 is about half the distance to the hand when the user stretches out his / her hand.

[0087] The position and orientation acquisition unit 1110 acquires the positions of the user's left hand 6050 and right hand 6060, and detects that the thumb and index finger of the left hand 6050 and right hand 6060 are each in an L-shape to form a rectangular frame shape. The selection unit 1120 can start a selection mode when the frame shape formed by the user's hand is detected. FIG. 6B shows an example in which the frame shape is formed in front of the user so as to be parallel to the XY plane, but the frame shape may be formed at a position shifted from the front of the user, or may be formed with an inclination with respect to the XY plane. Also, the frame shape may be formed with each side inclined with respect to the X-axis and the Y-axis.

[0088] Fig. 6(C) shows the state after the user's left hand 6050 and right hand 6060 have moved from the state shown in Fig. 6(B). The user's left hand 6050 moves to the upper left relative to the user's field of view while maintaining the distance in the depth direction. The user's right hand 6060 moves to the lower right relative to the user's field of view and also moves forward.

[0089] The selection unit 1120 sets a selection range 6070 based on the viewpoint of the user 7010, position information of the user's left hand 6050 and right hand 6060, and the number of virtual objects within the range surrounded by the left and right hands. In Fig. 6(C), when the user's left hand 6050 and right hand 6060 are projected onto a composite image in a three-dimensional space, there are four virtual objects present within a range obtained by expanding in the depth direction a rectangle formed by the thumb and index finger of the user's left hand 6050 and right hand 6060. The selection unit 1120 sets the selection range 6070 based on the positional relationship of the virtual objects and the positional relationship in the depth direction of the left and right hands.

[0090] A method for setting a selection range 6070 based on the positional relationship of virtual objects and the positional relationship of the left and right hands in the depth direction will be described with reference to Fig. 7(A). Fig. 7(A) is an elevation view showing the positional relationship in the depth direction between a user 7010 viewing the image of Fig. 6(C) and a virtual object. A division line 7050 is included between the user's left and right hands. Therefore, the selection unit 1120 sets the selection range 6070 so as to include the virtual object 6020 corresponding to the division line 7050, and changes the virtual object 6020 to a selected state.

[0091] Fig. 6(D) shows a state in which the user's right hand 6060 has moved forward from the state shown in Fig. 6(C). The selection unit 1120 updates the selection range 6070 and expands it forward. The updated selection range 6080 includes the virtual object 6020 and the virtual object 6030.

[0092] A method for setting the selection range 6080 based on the positional relationship of the virtual objects and the positional relationship of the left and right hands in the depth direction will be described with reference to FIG. 7(B). FIG. 7(B) is an elevation view showing the positional relationship in the depth direction between a user 7010 viewing the image of FIG. 6(D) and a virtual object. A division line 7040 and a division line 7050 are included between the user's left hand and right hand. Therefore, the selection unit 1120 sets the selection range 6080 so as to include the virtual object 6020 and the virtual object 6030 corresponding to the division line 7040 and the division line 7050, respectively. The selection unit 1120 changes the virtual object 6020 and the virtual object 6030 to a selected state.

[0093] Fig. 6(E) shows a state in which the amount of movement of the user's left hand 6050 and right hand 6060 in a predetermined time period has become equal to or less than a threshold value from the state shown in Fig. 6(D). When the selection range 6080 is set and the selection state of the virtual object is changed, the selection mode ends. When the selection mode ends, the selection range 6080 shown by the dotted line may be hidden.

[0094] In the third embodiment described above, the selection unit 1120 sets the selection range based on the relative positional relationship between the range in which the hand moves and the range in which the virtual object is placed in the depth direction. This allows the user to select a desired virtual object as intended from multiple virtual objects that are placed overlapping each other in the depth direction in a three-dimensional space.

[0095] [Variations] In each of the above embodiments, the imaging unit 1010 has one camera, but the imaging unit 1010 may be, for example, a stereo camera having two cameras corresponding to the left and right eyes of the user. When the imaging unit 1010 is a stereo camera, the selection range 6070 in Fig. 6(C) and the selection range 6080 in Fig. 6(D) may be set as a range visible by both cameras of the stereo camera.

[0096] The selection range 6070 and the selection range 6080 may be set as a range visible by the camera corresponding to the open eye or the dominant eye of the user's left or right eye. The open eye and the dominant eye can be acquired by known technology. The dominant eye may be set in advance or by the user.

[0097] In addition, although each of the above embodiments shows an example in which the selection unit 1120 performs the process of setting a selection range and changing the selection state of a virtual object only once, the process may be performed multiple times. The selection unit 1120 may set multiple selection ranges by repeating the process of setting a selection range multiple times. The selection unit 1120 changes the selection state of virtual objects included in the multiple selection ranges that have been set.

[0098] In addition, in each of the above embodiments, the selection unit 1120 switches the state of the virtual object included in the selection range from a deselected state to a selected state, but this is not limited to the above. The selection unit 1120 may switch the state of the virtual object included in the selection range from a selected state to a deselected state. For example, when the selection range is specified by a predetermined operation, such as when the selection range is specified by pointing the thumb outward or when the selection range is specified with the tip of the thumb, the selection unit 1120 switches the state of the virtual object in the selection range from a selected state to a deselected state. Note that the predetermined operation may be an operation using a controller.

[0099] Furthermore, when switching the state of a virtual object included in the selection range from a selected state to a deselected state, the selection unit 1120 changes the state of a virtual object not included in the selection range to a selected state. In addition, when a selection range includes a mixture of selected and deselected virtual objects, the selection unit 1120 may invert the selected and deselected states of the virtual objects in the selection range.

[0100] In each of the above embodiments, the selection unit 1120 determines the selection range by setting the selection range and changing the selection state of the virtual object, but the determined selection range may be moved by an operation from the user. For example, when an operation to move a frame indicating the determined selection range is received from the user, the selection unit 1120 moves the selection range and changes the selection state of the virtual object based on the selection range after the movement. By moving the determined selection range, the user can adjust the position of the selection range and easily select a desired virtual object. Note that, when moving the selection range, the selection unit 1120 may adjust the size of the selection range according to the distance from the user's position. For example, the selection unit 1120 may expand the selection range in a direction perpendicular to the depth direction (X-axis direction and Y-axis direction) as the distance from the user's position increases.

[0101] In each of the above embodiments, the selection unit 1120 sets a frustum-shaped range that is expanded in the direction in which the virtual object is placed with the user's eye position (viewpoint) as the reference position as the selection range, but the reference position is not limited to the user's eye position. This is because, if the user's eye position is set as the reference position, the frustum shape may not be expanded in the direction in which the virtual object is placed depending on the position of the user's hand that specifies the selection range.

[0102] For example, the selection unit 1120 may set the reference position to a position other than the eye position (such as the abdomen or chest) according to the position of the hand of the user who specifies the selection range. The selection unit 1120 can set, as the selection range, a frustum-shaped range that extends from the reference position determined according to the position of the hand of the user who specifies the selection range, through the position of the user's hand, in the direction in which the virtual object is placed.

[0103] Specifically, when a user specifies a selection range by making an L-shape with the thumbs and index fingers of both hands, the reference position can be a position where a straight line that passes through the midpoint of the bases of the thumbs and is approximately perpendicular to two planes specified by the thumbs and index fingers of each hand intersects with the user. The user can appropriately set the selection range by changing the reference position based on the position of the user's hand that specifies the selection range.

[0104] [Fourth embodiment] In the above-described embodiments, each unit constituting the information processing device 1100 shown in Fig. 1 is configured with hardware. In a fourth embodiment, part of the configuration of the information processing device 1100 may be configured with software. The information processing device 1100 according to this embodiment is a computer that realizes part of the operations described in the above-described embodiments by executing software and implements the remaining operations (functions) as hardware.

[0105] 8 is a block diagram showing an example of the hardware configuration of a computer applicable to the information processing device 1100. A CPU 8001 uses programs and data stored in a RAM 8002 and a ROM 8003 to control the entire computer and execute each process of the information processing device 1100 described in each of the above embodiments.

[0106] The RAM 8002 has an area for temporarily storing programs and data loaded from an external storage device 8007 or a storage medium drive 8008. The RAM 8002 has an area for temporarily storing data received from an external device via an I / F (interface) 8009. The external device is, for example, the display device 1000. The data received from the external device is, for example, a real space image and a display based on an operation from a user. The input value is generated by an input device of device 1000 .

[0107] The RAM 8002 also has a work area used when the CPU 8001 executes each process. That is, the RAM 8002 can provide various areas as appropriate. For example, the RAM 8002 also functions as the data storage unit 1150 shown in FIG.

[0108] The ROM 8003 is a non-volatile memory that stores the setting data and boot program of the computer.

[0109] The keyboard 8004 and the mouse 8005 are examples of operation input devices, and a computer user can input various instructions to the CPU 8001 by operating them.

[0110] The display unit 8006 is, for example, a CRT display or a liquid crystal display, and can display the results of processing by the CPU 8001 as images and characters. For example, the display unit 8006 can display messages for measuring the position and orientation of the display device 1000.

[0111] The external storage device 8007 is a large-capacity information storage device such as a hard disk drive, and stores an operating system (OS), programs for causing the CPU 8001 to execute various processes of the information processing device 1100, and data.

[0112] The programs stored in the external storage device 8007 include programs corresponding to the processes of the position and orientation acquisition unit 1110, the selection unit 1120, the image generation unit 1130, and the image synthesis unit 1140. The data stored in the external storage device 8007 includes the information described as known information and various setting information, in addition to the data of the virtual space. The programs and data saved in the external storage device 8007 are loaded into the RAM 8002 as appropriate under the control of the CPU 8001. The CPU 8001 executes each process of the information processing device 1100 by executing processes using the programs and data loaded into the RAM 8002. The external storage device 8007 may be used as the data storage unit 1150 shown in FIG. 1.

[0113] The storage medium drive 8008 reads out programs and data recorded on computer-readable storage media such as CD-ROMs or DVD-ROMs, and writes programs and data to these storage media. Note that some or all of the programs and data stored in the external storage device 8007 may be recorded on these storage media. The programs and data read by the storage medium drive 8008 from the storage media are output to the external storage device 8007 or the RAM 8002.

[0114] The I / F 8009 is an analog video port or a digital input / output port such as IEEE1394 for connecting the imaging unit 1010 of the display device 1000. The I / F 8009 may also be an Ethernet (registered trademark) port for outputting a composite image to the display unit 1020 of the display device 1000. Data received via the I / F 8009 is input to the RAM 8002 or the external storage device 8007. When a sensor system is used for the position and orientation acquisition unit 1110 to acquire position and orientation information, the I / F 8009 is used as an interface for connecting the sensor system. The bus 8010 connects the units illustrated in FIG. 8 to each other.

[0115] [Other embodiments] The present disclosure relates to a method for directly or remotely programming a software program into a system or device. The program code is supplied from the computer of the system or device, and the computer of the system or device reads and executes the supplied program code to achieve the functions of each of the above embodiments. The program supplied to the system or device is a program for executing the processes corresponding to the flowchart described in FIG.

[0116] The functions in each of the above embodiments may be realized by a computer executing a read program, or may be realized in cooperation with an OS running on a computer based on instructions from the program. In this case, the functions in each of the embodiments are realized by the OS executing a part or all of the functions.

[0117] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0118] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A display control means for controlling the display of a virtual object so that the virtual object is placed in a three-dimensional space within a user's field of view; a setting means for setting a selection range in the three-dimensional space based on a plurality of positions at different distances from the user, the positions being specified using an operating object at the position of the user's hand; a selection means for switching a state of the virtual object between a selected state and a deselected state based on the selection range; 13. An information processing device comprising: (Configuration 2) The selection area is frustum shaped. 2. The information processing device according to configuration 1. (Configuration 3) The operation object is a hand of the user, The setting means sets the selection range based on the shape of the user's hand. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The setting means sets the selection range in the shape of a truncated cone or a truncated pyramid based on the shape formed by the user's finger. 4. The information processing device according to configuration 3. (Configuration 5) The selection means is determining that the virtual object is included in the selection range when a part of the virtual object is included in the selection range, when the entire virtual object is included in the selection range, or when the center of gravity of the virtual object is included in the selection range; A state of the virtual object included in the selection range is switched between a selected state and a deselected state. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The setting means sets the selection range based on a relative positional relationship between the operating body and the virtual object. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The setting means associates the virtual object with a range in the depth direction in which the operating body moves, and sets the virtual object corresponding to the range in the depth direction of the plurality of positions designated by the operating body. Set the selection to include the body 7. The information processing device according to configuration 6, (Configuration 8) The setting means associates the virtual objects with the range in the depth direction in which the operating body moves at equal intervals or at intervals based on the intervals between the virtual objects. The information processing device according to configuration 7, (Configuration 9) When performing display control of a plurality of virtual objects, the plurality of virtual objects are associated with a range in the depth direction in which the operating body moves at intervals equal to or greater than a predetermined threshold. The information processing device according to configuration 8, (Configuration 10) The selection means switches a state of the virtual object included in the selection range to a deselected state when the selection range is set by a predetermined operation with the operating object. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) When the selection range is set by the predetermined operation, the selection means further switches a state of the virtual objects not included in the selection range to a selected state. 11. The information processing device according to configuration 10. (Configuration 12) The plurality of positions are designated by one hand of the user. 12. The information processing device according to any one of configurations 1 to 11. (Configuration 13) The setting means sets a plurality of selection ranges. 13. The information processing device according to any one of configurations 1 to 12. (Configuration 14) The setting means moves the selection range by an operation using the operating object, The selection means switches a state of the virtual object between a selected state and a deselected state based on the selection range after the movement. 14. The information processing device according to any one of configurations 1 to 13. (Configuration 15) When moving the selection range, the setting means adjusts the size of the selection range in accordance with the distance from the user's position to the selection range. 15. The information processing device according to configuration 14. (Configuration 16) The setting means sets, as the selection range, a frustum-shaped range extending from an eye position of the user as a reference position through the plurality of positions in a direction in which the virtual object is disposed. 16. The information processing device according to any one of configurations 1 to 15. (Configuration 17) The setting means sets, as the selection range, a frustum-shaped range extending from a reference position based on the plurality of positions designated by the user's hand through the plurality of positions in a direction in which the virtual object is placed. 16. The information processing device according to any one of configurations 1 to 15. (Configuration 18) The three-dimensional space image is a composite image in which the virtual object is placed on a real space image captured by a stereo camera. 18. The information processing device according to any one of configurations 1 to 17. (method) a display control step of controlling the display of a virtual object so that the virtual object is placed in a three-dimensional space that is within the user's field of view; a setting step of setting a selection range in the three-dimensional space based on a plurality of positions at different distances from the user, the positions being specified using an operating object at the position of the user's hand; a selection step of switching a state of the virtual object between a selected state and a deselected state based on the selection range; An information processing method comprising causing a computer to execute the steps of: (program) 19. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 18. [Explanation of symbols]

[0119] 1100: information processing device, 1130: image generating unit, 1120: selection unit, 8001: CPU

Claims

1. a display control means for controlling the display of a plurality of virtual objects so that they are arranged in a three-dimensional space within the user's field of view; a determination means for determining whether a selection mode for switching the states of the plurality of virtual objects between a selected state and a deselected state has been started; a setting means for setting a selection range in the three-dimensional space based on a plurality of positions at different depth distances from the user, the positions being designated using an operating object at the position of the user's hand when the selection mode is started; a selection means for switching the state of a virtual object included in the selection range among the plurality of virtual objects to a selected state and switching the state of a virtual object not included in the selection range to a deselected state; An information processing device comprising:

2. The selected area is frustum-shaped 2. The information processing apparatus according to claim 1, wherein:

3. the operation object is the user's hand, The setting means sets the selection range based on the shape of the user's hand.

2. The information processing apparatus according to claim 1, wherein:

4. The setting means sets the selection range in the shape of a truncated cone or a truncated pyramid based on the shape formed by the user's finger.

4. The information processing apparatus according to claim 3,

5. The selection means determining that the virtual object is included in the selection range when a part of the virtual object is included in the selection range, when the entire virtual object is included in the selection range, or when the center of gravity of the virtual object is included in the selection range; The state of the virtual object included in the selection range is switched between a selected state and a deselected state.

2. The information processing apparatus according to claim 1, wherein:

6. The setting means sets the selection range based on a relative positional relationship between the operating body and the virtual object.

2. The information processing apparatus according to claim 1, wherein:

7. The setting means associates the virtual object with a range in the depth direction in which the operating body moves, and sets the selection range so as to include the virtual object corresponding to the range in the depth direction of the plurality of positions specified by the operating body.

7. The information processing apparatus according to claim 6, wherein:

8. The setting means associates the virtual objects with the range in the depth direction in which the operating body moves at equal intervals or at intervals based on the intervals between the virtual objects.

8. The information processing apparatus according to claim 7, wherein:

9. When controlling the display of a plurality of virtual objects, the plurality of virtual objects are associated with a range in the depth direction in which the operating body moves at intervals equal to or greater than a predetermined threshold.

9. The information processing apparatus according to claim 8, wherein:

10. When the selection range is set by a predetermined operation using the operating object, the selection means switches the state of the virtual object included in the selection range to a deselected state.

2. The information processing apparatus according to claim 1, wherein:

11. When the selection range is set by the predetermined operation, the selection means further switches the state of the virtual objects not included in the selection range to a selected state.

11. The information processing apparatus according to claim 10,

12. The plurality of positions are designated by one hand of the user.

2. The information processing apparatus according to claim 1, wherein:

13. The setting means sets a plurality of selection ranges.

2. The information processing apparatus according to claim 1, wherein:

14. the setting means moves the selection range by operation of the operating object; The selection means switches the state of the virtual object between a selected state and a deselected state based on the selection range after the movement.

2. The information processing apparatus according to claim 1, wherein:

15. When moving the selection range, the setting means adjusts the size of the selection range according to the distance from the user's position to the selection range.

15. The information processing apparatus according to claim 14,

16. The setting means sets, as the selection range, a frustum-shaped range extending from the reference position, passing through the plurality of positions, in a direction in which the virtual object is placed, using the position of the user's eyes as a reference position.

2. The information processing apparatus according to claim 1, wherein:

17. The setting means sets, as the selection range, a frustum-shaped range extending from a reference position based on the plurality of positions designated by the user's hand, through the plurality of positions, in a direction in which the virtual object is located.

2. The information processing apparatus according to claim 1, wherein:

18. The image of the three-dimensional space is a composite image in which the virtual object is placed on a real space image captured by a stereo camera.

2. The information processing apparatus according to claim 1, wherein:

19. The selection mode is initiated when: (1) the distance between the user's left and right hands is equal to or less than a first threshold; (2) the amount of movement of the user's hands for a first predetermined time is equal to or less than a second threshold; (3) the distance between the user's left and right hands is equal to or less than a third threshold for a second predetermined time; or (4) the user changes the shape of their hands to a predetermined shape.

2. The information processing apparatus according to claim 1, wherein:

20. The selection mode is initiated by the user's operation from an input device, or by the user's gaze or voice instruction.

2. The information processing apparatus according to claim 1, wherein:

21. The selection range has a truncated cone shape with the user side as the upper surface, the upper surface of the selection range being included in a first surface defined based on a first position specified using the operating body, and the bottom surface of the selection range being defined based on a second position that is farther from the user in the depth direction than the first position, and being included in a second surface opposing the first surface.

2. The information processing apparatus according to claim 1, wherein:

22. a display control step of controlling the display of a plurality of virtual objects so that they are arranged in a three-dimensional space within the user's field of view; a determination step of determining whether a selection mode for switching the states of the plurality of virtual objects between a selected state and a deselected state has been initiated; a setting step of setting a selection range in the three-dimensional space based on a plurality of positions at different depth distances from the user, the positions being specified using an operating object at the position of the user's hand when the selection mode is started; a selection step of switching a state of a virtual object included in the selection range out of the plurality of virtual objects to a selected state and switching a state of a virtual object not included in the selection range to a deselected state; An information processing method characterized by causing a computer to execute the above.

23. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 21.

Images