Information processing device, method for controlling information processing device, and program

Abstract

To provide an information processing device that enables easy selection of a virtual object distant from a user or a small virtual object.SOLUTION: An information processing device 1000 includes a CPU 2010 that determines whether there is an instruction of a user performing an operation of selecting a virtual object 5010 with a three-dimensional pointer 5000 in a space of composite reality or a space of virtual reality, and when the virtual object 5010 is not included in a selected area 5050 set in the vicinity of a center 5020 of the three-dimensional pointer 5000 in a case in which there is the instruction, gradually increases the size of the selected area 5050 until the virtual object 5010 is included in the selected area 5050, and fixes the virtual object 5010 included in the selected area 5050 as the virtual object selected by the user with the three-dimensional pointer 5000.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] In recent years, development of systems such as VR (Virtual Reality), AR (Augmented Reality), and MR (Mixed Reality) has progressed. VR, AR, and MR allow users to experience the sensation of digitally created images or CG (Computer Graphics) actually being there. VR is a system that creates a virtual space that is different from reality, creating an environment that feels as if it were real. AR is a system that creates an environment in which objects created using CG are reflected in the real world. MR is a system that creates an environment that combines and fuses a virtual space created using CG with the real world.

[0003] Conventionally, in environments created by these systems, a user may select an object by pointing a pointer from his / her hand, a controller, etc. onto the object to be selected to display the pointer. In this regard, for example, Patent Document 1 discloses a technology for determining the arrangement of an object and dynamically changing the size of the pointer. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6904973 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology disclosed in Patent Document 1 reduces the size of the pointer when multiple objects are placed close to each other, so that the user can select each object individually. Therefore, if the object to be selected is far away or small, the user has difficulty in aligning the pointer with the object due to the effects of camera shake, etc.

[0006] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an information processing device that enables a user to easily select a virtual object that is far from the user or a small virtual object. [Means for solving the problem]

[0007] In order to achieve the above object, the information processing device of the present invention is characterized by comprising: a determination means for determining whether or not a user has instructed to select a virtual object with a pointer in a mixed reality space or a virtual reality space; an enlargement means for gradually increasing a size of the selection area until the virtual object is included in the selection area when the user has instructed to do so and a determination means for determining that the virtual object included in the selection area is the virtual object selected by the user with the pointer. Effect of the Invention

[0008] According to the present invention, a virtual object that is far away from the user or a small virtual object can be easily selected. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a functional configuration of a system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing device according to the first embodiment. [Diagram 3]1 is a flowchart of a process performed by the information processing device according to the first embodiment to generate a compound spatial image and to output the generated compound spatial image to an HMD (Head Mounted Display) or an image display device. [Figure 4] 4 is a flowchart of a selection operation process of the information processing device according to the first embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of an operation performed when a user selects a virtual object with a three-dimensional pointer in the first embodiment. [Figure 6] FIG. 4 is a diagram illustrating the size of a selection region in the first embodiment. [Figure 7] 13 is a flowchart of a process performed by an information processing device according to a second embodiment to generate a compound spatial image and output the generated compound spatial image to an HMD or an image display device. [Figure 8] 13 is a flowchart of a normal selection operation process of an information processing device according to a second embodiment. [Figure 9] 13 is a flowchart of a list selection operation process of the information processing device according to the second embodiment. [Figure 10] FIG. 11 is a diagram showing an example of an operation performed when a user selects a virtual object with a three-dimensional pointer in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments. For example, each part constituting the present invention can be replaced with any configuration that can exert a similar function. In addition, any configuration may be added. In addition, any two or more configurations (features) of each embodiment can be combined.

[0011] First Embodiment Hereinafter, the first embodiment will be described with reference to Figs. 1 to 6. Fig. 1 is a block diagram showing an example of a functional configuration of a system 100 in the first embodiment. As shown in Fig. 1, the system 100 has an information processing device 1000, an HMD 1100, and an image display device 1200. The information processing device 1000 is, for example, configured by the main body of a desktop PC, which is a general-purpose computer. However, the information processing device 1000 is not limited thereto, and may be, for example, a general-purpose computer such as a laptop PC, a tablet PC, or a smartphone. In addition, if the HMD 1100 has the functional configuration of the information processing device 1000 described later, it can also be treated as the information processing device 1000 of the present invention.

[0012] The information processing device 1000 includes a captured image acquisition unit 1010, a position and orientation measurement unit 1020, a virtual data storage unit 1030, a three-dimensional pointer position and orientation acquisition unit 1040, an object selection determination unit 1050, a virtual space generation unit 1060, and a composite space image generation unit 1070. The HMD 1100 is an example of a head-mounted display device, and includes an image capture unit 1110 and an image display unit 1120. The information processing device 1000 and the HMD 1100 are connected to each other so that data communication is possible. The connection between the information processing device 1000 and the HMD 1100 may be wired or wireless. This point is also true for the connection between the information processing device 1000 and the image display device 1200. The image display device 1200 is, for example, configured of a liquid crystal display.

[0013] Here, first, the hardware of the information processing device 1000 will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device 1000. As shown in FIG. 2, the information processing device 1000 has a bus 2000, a CPU 2010, a ROM 2020, a RAM 2030, an input I / F 2040, and an output I / F 2050. The CPU 2010 comprehensively controls each device connected via the bus 2000. The CPU 2010 reads and executes each processing program stored in a ROM 2020 that is a read-only memory. An operating system (OS), each processing program, device drivers, and the like are stored in the ROM 2020, temporarily stored in the RAM 2030, and appropriately executed by the CPU 2010. The input I / F 2040 inputs a signal from an external device (for example, an imaging device, an operating device, etc.) to the information processing device 1000 as an input signal in a format that can be processed by the information processing device 1000. The output I / F 2050 outputs a signal from the information processing device 1000 to an external device (eg, a display device) as an output signal in a format that can be processed by the external device.

[0014] Returning to the description of FIG. 1, the functional configuration of the information processing device 1000 will be described next. The captured image acquisition unit 1010 acquires an image of real space (hereinafter, referred to as a "real space image") captured by the image capturing unit 1110 of the HMD 1100, and inputs the image to the position and orientation measurement unit 1020 and the compound space image generation unit 1070. The position and orientation measurement unit 1020 performs image processing using the real space image acquired by the captured image acquisition unit 1010, and measures the position and orientation by extracting characteristic information such as points and lines in the real space image. The position and orientation measurement unit 1020 inputs the position and orientation information acquired by such measurement to the compound space image generation unit 1070. In the first embodiment, as described above, the position and orientation measurement unit 1020 measures the position and orientation using image processing, but may measure the position and orientation using infrared light, may measure the position and orientation using ultrasonic waves, or may measure the position and orientation using a magnetic sensor. Furthermore, the position and orientation measuring unit 1020 may measure the position and orientation using a depth sensor, or may measure the position and orientation mechanically.

[0015] The virtual data holding unit 1030 holds data on the virtual space, such as data on virtual objects constituting the virtual space and data on a light source irradiated in the virtual space, and inputs the data to the object selection determination unit 1050 and the virtual space generation unit 1060. The data on the virtual objects constituting the virtual space includes shape information on the virtual objects and position and orientation information on the virtual objects. The three-dimensional pointer position and orientation acquisition unit 1040 acquires position and orientation information of a three-dimensional pointer (pointer) for selecting a virtual object, and inputs the information to the object selection determination unit 1050. The three-dimensional pointer is a tool used by the user to select a virtual object. In the first embodiment, the information processing device 1000 uses a laser-shaped virtual object that the user can move arbitrarily as the three-dimensional pointer, but is not limited to this. For example, the information processing device 1000 may use a gaze point as the three-dimensional pointer. In addition, the information processing device 1000 may use a part of the user's body, such as a hand, capable of acquiring position and orientation information in the virtual space, as the three-dimensional pointer.

[0016] The object selection determination unit 1050 acquires shape information and position and orientation information of a virtual object from the virtual space data stored in the virtual data storage unit 1030. Furthermore, the object selection determination unit 1050 determines whether or not the selection of a virtual object has been confirmed based on the shape information and position and orientation information of the virtual object, as well as the position and orientation information of the three-dimensional pointer acquired by the three-dimensional pointer position and orientation acquisition unit 1040. The object selection determination unit 1050 inputs the shape information and position and orientation information of a virtual object determined to have been selected as a confirmed virtual object, and the position and orientation information of the three-dimensional pointer, to the virtual space generation unit 1060.

[0017] The virtual space generation unit 1060 generates a virtual space based on the virtual space data held by the virtual data holding unit 1030. Furthermore, the virtual space generation unit 1060 inputs the generated virtual space to the compound space image generation unit 1070. Note that the technology for generating a virtual space visible from a predetermined position is a known fact, so a detailed description thereof will be omitted. The compound space image generation unit 1070 generates a virtual space image based on the virtual space generated by the virtual space generation unit 1060 and the position and orientation information of the position and orientation measurement unit 1020. Furthermore, the compound space image generation unit 1070 superimposes the generated virtual space image on the real space image acquired by the captured image acquisition unit 1010 to generate an image of the compound space (hereinafter referred to as a "compound space image"). In this way, the compound space image generated by the compound space image generation unit 1070 is an image of mixed reality (MR). In other words, the compound space is a space of mixed reality (MR). Thereafter, the compound spatial image generating unit 1070 outputs the generated compound spatial image to the image display unit 1120 of the HMD 1200. Note that the output destination of the compound spatial image is not limited to the HMD 1200, and may be any device capable of displaying the compound spatial image, such as a tablet, a smartphone, or another portable terminal capable of displaying the compound spatial image. The compound spatial image generating unit 1070 may also output the generated compound spatial image to the image display device 1200 at the same time.

[0018] Each functional unit of the information processing device 1000 is realized by a CPU 2010 (computer) loading a program stored in a ROM 2020 into a RAM 2030 and executing processing in accordance with each flowchart shown in Figures 3, 4, and 7 to 9, which will be described later. When configuring hardware in the information processing device 1000 as an alternative to software processing using the CPU 2010, it is sufficient to configure a calculation unit or circuit corresponding to the processing of each functional unit of the information processing device 1000.

[0019] Next, the functional configuration of the HMD 1100 will be described. The HMD 1100 is a display device configured with a liquid crystal screen or the like (hereinafter referred to as a "screen"). Screens for the right and left eyes are provided, and the right and left eyes are attached to the HMD 1100 so as to be located in front of the right and left eyes of a user wearing the HMD 1100 on the head. In addition, images with parallax are stereo-displayed on each of the left and right screens. The image capturing unit 1110 captures the real space reflected in the HMD 1100, and outputs the captured image to the information processing device 1000 as a real space image. The image display unit 1120 displays a compound space image generated by the information processing device 1000. As described above, in the first embodiment, the HMD 1100 is a video see-through type HMD (that is, a display device that displays a compound space image generated based on an image captured by an imaging device). However, the HMD 1100 may be an optical see-through HMD (that is, one that displays a virtual space image superimposed on a display medium that allows observation through the real space) or the like.

[0020] 3 is a flowchart of a process performed by the information processing device 1000 to generate a compound space image and output the generated compound space image to the HMD 1100 or the image display device 1200. As described above, the process (method of controlling the information processing device) of FIG. 3 is realized by the CPU 2010 (computer) expanding a program stored in the ROM 2020 into the RAM 2030 and executing it. In step S3010, the position and orientation measurement unit 1020 acquires position and orientation information by measuring the position and orientation using a real space image of the image capturing unit 1110 of the HMD 110 (i.e., a real space image acquired by the captured image acquisition unit 1010). Then, the process proceeds to step S3020. In step S3020, the three-dimensional pointer position and orientation acquisition unit 1040 acquires position and orientation information of the three-dimensional pointer. Then, the process proceeds to step S3030.

[0021] In step S3030, the object selection determination unit 1050 determines whether the selection of the virtual object by the three-dimensional pointer is confirmed. If the object selection determination unit 1050 determines that the selection of the virtual object by the three-dimensional pointer is confirmed, the process proceeds to step S3040. On the other hand, if the object selection determination unit 1050 determines that the selection of the virtual object by the three-dimensional pointer is not confirmed, the process proceeds to step S3050. In step S3040, the CPU 2010 updates the information in the virtual data storage unit 1030 so as to change the absolute position of the virtual object whose selection has been confirmed to the position indicated by the position information of the three-dimensional pointer acquired in step S3020. Then, the process proceeds to step S3060.

[0022] In step S3050, a selection operation process is performed. In the selection operation process, a process is performed for an operation of the user selecting a virtual object based on the position and orientation information and shape information of the virtual object in the virtual data storage unit 1030 and the position and orientation information of the three-dimensional pointer acquired in step S3020. A detailed flow of the selection operation process will be described later. In step S3060, the virtual space generation unit 1060 generates a virtual space using the virtual space data stored in the virtual data storage unit 1030. Then, the process proceeds to step S3070.

[0023] In step S3070, the compound space image generation unit 1070 generates a virtual space image based on the virtual space generated by the virtual space generation unit 1060 and the position and orientation information of the position and orientation measurement unit 1020. Furthermore, the compound space image generation unit 1070 generates a compound space image by superimposing the generated virtual space image on the real space image acquired by the captured image acquisition unit 1010. Furthermore, the compound space image generation unit 1070 outputs the generated compound space image to the image display unit 1120 of the HMD 1100 or the image display device 1200. Then, the process proceeds to step S3080.

[0024] In step S3080, the CPU 2010 determines whether to end the process in Fig. 3. If an end instruction is input by the user or if the end conditions are met, the CPU 2010 determines to end the process in Fig. 3. This ends the process in Fig. 3. On the other hand, if an end instruction is not input by the user and the end conditions are not met, the CPU 2010 determines not to end the process in Fig. 3. This returns the process to step S3010.

[0025] FIG. 4 is a flowchart of the selection operation process of step S3050. In step S4010, the CPU 2010 (determination means) determines whether or not the user has instructed to perform an operation to select a virtual object (determination step). In the first embodiment, the user instructs to perform an operation to select a virtual object by keeping the center position of the three-dimensional pointer within a predetermined range for a predetermined time. However, the user may instruct to perform an operation to select a virtual object by operating a button on an operating device such as a controller, or by having the information processing device 1000 recognize the user's hand gesture. In this way, the user can reliably give an instruction to perform an operation to select a virtual object to the information processing device 1000. Note that these points are similar to those in the second embodiment described later. If the CPU 2010 determines that the user has instructed to perform an operation to select a virtual object, the process proceeds to step S4020. On the other hand, if the CPU 2010 determines that the user has not instructed to perform an operation to select a virtual object, the process proceeds to step S4030.

[0026] Note that, even if the user instructs to cancel the operation of selecting a virtual object, the process proceeds to step S4030. In the first embodiment, the instruction to cancel the operation of selecting a virtual object is given when the user moves the position of the three-dimensional pointer out of the above-mentioned predetermined range. However, the user may give the instruction to cancel the operation of selecting a virtual object by operating a button on an operating device such as a controller, or by having the information processing device 1000 recognize a hand gesture by the user.

[0027] In step S4030, CPU 2010 resets the size of the selection area near the three-dimensional pointer to its initial state. The selection area will be described later. Then, the process of FIG. 4 ends, and the process proceeds to step S3060. In step S4020, CPU 2010 determines whether or not the virtual object is within the selection area near the three-dimensional pointer. This determination is made based on the presence or absence of interference between the three-dimensional pointer and the virtual object, based on the position and orientation information and shape information of the virtual object in virtual data storage unit 1030, and the position and orientation information of the three-dimensional pointer acquired in step S3020. The technology for determining the presence or absence of interference is a known fact, so a detailed description thereof will be omitted.

[0028] The process of FIG. 4 will be described below with reference to FIG. 5. FIG. 5 is a diagram showing an example of an operation when a user selects a virtual object 5010 with a three-dimensional pointer 5000. FIG. 5(a) is a diagram showing a case where a user moves a center 5020 of the three-dimensional pointer 5000 close to the virtual object 5010 in order to select the virtual object 5010. A line 5030 represents a direction pointing from a user's hand 5040 to the center 5020 of the three-dimensional pointer 5000, and may or may not be displayed as a ray as if a ray were coming from the user's hand 5040. In the case of FIG. 5(a), a selection area 5050 near the center 5020 of the three-dimensional pointer 5000 does not include any virtual object. In such a case, in step S4020, it is determined that a virtual object is not within the selection area near the three-dimensional pointer, and the process of FIG. 4 proceeds from step S4020 to step S4040. This point is similar to the case of FIG. 5(b) described later. The state of the selection area 5050 shown in FIG. 5(a) is an example of the initial state of the selection area.

[0029] In contrast, in the case of FIG. 5(c), a part of the virtual object 5010 is included in the selection area 5050 near the center 5020 of the three-dimensional pointer 5000. In such a case, in step S4020, it is determined that the virtual object is within the selection area near the three-dimensional pointer, and the process of FIG. 4 proceeds from step S4020 to step S4050. In step S4050, the CPU 2010 confirms the selection of the virtual object. In the case of FIG. 5(c), the selection of the virtual object 5010 is confirmed. Then, the process of FIG. 4 ends, and the process proceeds to the above step S3060. In step S4040, the CPU 2010 (enlarging means) enlarges the size of the selection area near the three-dimensional pointer by one step (enlarging process). Then, the process of FIG. 4 ends, and the process proceeds to the above step S3060. The user may increase the size of the selection area near the three-dimensional pointer step by step by operating a button on an operating device such as a controller, or by having the information processing device 1000 recognize the user's hand gesture. In this case, the user can gradually increase the size of the selection area near the three-dimensional pointer by his / her own operation.

[0030] FIG. 5B is a diagram showing a state in which the size of a selection area 5050 near the center 5020 of the three-dimensional pointer 5000 is expanded by one step from the state in FIG. 5A. Hereinafter, a case will be described in which, in the state in FIG. 5B, a user continues to instruct the operation of selecting a virtual object. In this case, the process proceeds from step S3080 to step S3010. Furthermore, the process proceeds from step S3010 to step S3020 and then to step S3030, but since the selection of the virtual object has not yet been confirmed, the process proceeds to step S3050. As a result, the process proceeds to step S4010, but since the user continues to instruct the operation of selecting a virtual object, the process proceeds to step S4020. In step S4020, in the state in FIG. 5B, it is determined that the virtual object is not within the selection area near the three-dimensional pointer, and the process proceeds to step S4040.

[0031] In step S4040, the size of the selection area near the three-dimensional pointer is enlarged by one step. FIG. 5(c) is a diagram showing the state when the size of the selection area 5050 near the center 5020 of the three-dimensional pointer 5000 is enlarged by one step from the state shown in FIG. 5(b). The rate at which the size of the selection area 5050 is enlarged may be constant at each step, or may be changed by increasing or decreasing the size stepwise. The rate at which the size of the selection area 5050 is enlarged may be changed so that the rate increases the longer the time that the center 5020 of the three-dimensional pointer 5000 remains in the above-mentioned predetermined range. In this case, the user can change the rate at which the size of the selection area 5050 is enlarged by his / her own operation. Then, the process proceeds to step S3060. At this time, the instruction to perform the operation of selecting the virtual object by the user is still ongoing, so the process proceeds to step S4020 again as described above. 5(c), as described above, it is determined that the virtual object is within the selection area near the three-dimensional pointer, and the process proceeds to step S4050. In step S4050, the CPU 2010 (confirmation step) confirms the selection of the virtual object (confirmation step). This confirms the selection of the virtual object 5010. In this manner, the user selects the virtual object 5010.

[0032] Here, a method of defining the size of the selection area 5050 will be described. FIG. 6 is a diagram for explaining the size of the selection area 5050. In FIG. 6, the same reference numerals are used for the same parts as in FIG. 5. The method of defining the size of the selection area 5050 includes a method of defining it in a spherical shape and a method of defining it in a conical shape. In the method of defining the size of the selection area 5050 in a spherical shape, the size is defined by the radius 6010 of the sphere. The sphere is the surface of a sphere centered on the center 5020 of the three-dimensional pointer 5000, which is the point where the line 5030 extending from the position of the user's hand 5040, which is the starting point of the three-dimensional pointer 5000, hits the object. In addition, even when the center 5020 of the three-dimensional pointer is placed in space by a controller or a hand gesture, the size is similarly defined by the radius of the sphere centered on the center 5020 of the three-dimensional pointer 5000. In other words, in the method of defining the size of the selection area 5050 in a spherical shape, the size is defined by the distance from the center 5020 of the three-dimensional pointer 5000. On the other hand, in a method of defining the size of the selection area 5050 in a cone shape, a line 5030 from the position of the user's hand 5040, which is the starting point of the three-dimensional pointer 5000, to the center 5020 of the three-dimensional pointer is set as the rotation axis of the cone. In addition, in a method of defining the size of the selection area 5050 in a cone shape, the size is defined by an angle 6030 between the generating line 6020 of the cone and the line 5030.

[0033] As described above, according to the first embodiment, when a user instructs the information processing device 1000 to perform an operation to select a virtual object, the information processing device 1000 gradually increases the size of the selection region 5050 near the center 5020 of the three-dimensional pointer 5000 in the mixed reality (MR) space. This allows the user to select a virtual object without precisely aligning the center 5020 of the three-dimensional pointer 5000 with the virtual object, making it easy to select a virtual object that is far from the user or a small virtual object.

[0034] The first embodiment is an example in which the present invention is applied to an MR system that displays a compound space image in which a virtual space image is superimposed on a real space image. However, the present invention is not limited to this, and can be applied to, for example, a VR system that displays only a virtual space image. That is, when a user instructs the information processing device 1000 to perform an operation to select a virtual object, the information processing device 1000 gradually increases the size of a selection area 5050 near a center 5020 of a three-dimensional pointer 5000 in a virtual reality (VR) space. In this case, the compound space image generating unit 1070 of the information processing device 1000 may display the virtual space image as it is on the image display unit 1120 of the HMD 1100 without superimposing the virtual space image on the real space image acquired by the image capturing unit 1110 of the HMD 1100. In addition, in such a VR system, when the position and orientation are measured using a sensor, the image capturing unit 1110 of the HMD 1100 may not be required.

[0035] Also, different from the first embodiment, the information processing device 1000 may gradually increase the size of the selection region 5050 near the center 5020 of the three-dimensional pointer 5000 before the user instructs the operation to select a virtual object. In this case, the user can select the virtual object 5010 by instructing the operation to select the virtual object at a timing when the selection region 5050 includes a part or the whole of the virtual object 5010.

[0036] <Second embodiment> Hereinafter, the second embodiment will be described with reference to Figs. 7 to 10. Here, the differences from the first embodiment will be mainly described. In the first embodiment, a configuration was described in which, when a user instructs to perform an operation to select a virtual object, the size of the selection area 5050 near the center 5020 of the three-dimensional pointer 5000 gradually increases. In the second embodiment, a configuration will be described in which all virtual objects included in the selection area 5050 near the center 5020 of the three-dimensional pointer 5000 are displayed in a list in a subwindow, and a virtual object is selected from the subwindow. The functional configuration of the system 100 and the hardware configuration of the information processing device 1000 in the second embodiment are the same as those in Figs. 1 and 2 of the first embodiment, and therefore detailed description thereof will be omitted.

[0037] 7 is a flowchart of the process performed by the information processing device 1000 to generate a compound space image and output the generated compound space image to the HMD 1100 or the image display device 1200. As described above, the process in FIG. 7 is realized by the CPU 2010 loading a program stored in the ROM 2020 into the RAM 2030 and executing it. Steps S7010 to S7040 are the same as steps S3010 to S3040 in FIG. 3, and therefore detailed description thereof will be omitted. In step S7050, the CPU 2010 determines whether a subwindow is being displayed. The subwindow will be described later.

[0038] If CPU 2010 determines that a subwindow is not being displayed, the process proceeds to step S7060. On the other hand, if CPU 2010 determines that a subwindow is being displayed, the process proceeds to step S7070. In step S7060, normal selection operation processing is performed. In step S7070, list selection operation processing is performed. FIG. 8 is a flowchart of the normal selection operation processing in step S7060. FIG. 9 is a flowchart of the list selection operation processing in step S7070.

[0039] As shown in Fig. 8, in the normal selection operation process, a process is performed for a user's operation of selecting a virtual object based on the position and orientation information and shape information of the virtual object in the virtual data storage unit 1030 and the position and orientation information of the three-dimensional pointer acquired in step S7020. In step S8010, the CPU 2010 (determination means) determines whether or not the user has instructed to perform an operation of selecting a virtual object. If the CPU 2010 determines that the user has instructed to perform an operation of selecting a virtual object, the process proceeds to step S8020. On the other hand, if the CPU 2010 determines that the user has not instructed to perform an operation of selecting a virtual object, the process in Fig. 8 ends and the process proceeds to step S7080, which will be described later. The user instructs to perform an operation of selecting a virtual object in the same manner as in the first embodiment.

[0040] Step S8020 is the same as step S4020 in FIG. 4, so a detailed description thereof will be omitted. If the CPU 2010 determines in step S8020 that the virtual object is within the selection area near the three-dimensional pointer, the process proceeds to step S8030. On the other hand, if the CPU 2010 determines that the virtual object is not within the selection area near the three-dimensional pointer, the process of FIG. 8 ends, and the process proceeds to step S7080, which will be described later. Note that, if the CPU 2010 (enlarging means) determines that the virtual object is not within the selection area near the three-dimensional pointer, it may gradually increase the size of the selection area until the virtual object is included in the selection area, as in the first embodiment. In this case, after it is determined that the virtual object is not within the selection area near the three-dimensional pointer, one or more virtual objects can be included in the selection area near the three-dimensional pointer without the user moving the three-dimensional pointer.

[0041] In step S8030, it is determined whether the number of virtual objects in the selection area near the three-dimensional pointer is singular. Hereinafter, the processing of FIG. 8 will be described with reference to FIG. 10. FIG. 10 is a diagram showing an example of an operation when the user selects a virtual object 10020 with the three-dimensional pointer 5000. In FIG. 10, the same reference numerals are used for the same parts as in FIG. 5. FIG. 10(a) is a diagram showing a case where the user moves the center 5020 of the three-dimensional pointer 5000 close to the virtual object 10020 in order to select the virtual object 10020 with the three-dimensional pointer 5000. In the case of FIG. 10(a), the selection area 5050 near the center 5020 of the three-dimensional pointer 5000 includes parts of each of the three virtual objects 10010, 10020, and 10030. In such a case, in step S8030, it is determined that the number of virtual objects in the selection area near the three-dimensional pointer is not singular, and the processing of FIG. 8 proceeds from step S8030 to step S8040. In the case of FIG. 10( a ), the selection area 5050 in the vicinity of the center 5020 of the three-dimensional pointer 5000 does not include the virtual object 10040 .

[0042] On the other hand, in the case of FIG. 10(b), the selection area 5050 near the center 5020 of the three-dimensional pointer 5000 includes only a part of one virtual object 10020. In such a case, in step S8030, it is determined that the number of virtual objects in the selection area near the three-dimensional pointer is one, and the process of FIG. 8 proceeds from step S8030 to step S8050. In step S8050, the CPU 2010 confirms the selection of the virtual object. As a result, if the number of virtual objects in the selection area near the three-dimensional pointer is one, the selection of the virtual object is confirmed. Therefore, in the case of FIG. 10(b), the selection of the virtual object 10020 is confirmed. In this manner, the user selects the virtual object 10020. Then, the process of FIG. 8 ends, and the process proceeds to step S7080, which will be described later.

[0043] The size of the selection area 5050 may be set in advance, or may be changed based on the distance from the user's hand 5040, which is the starting point of the three-dimensional pointer 5000, to the center 5020 of the three-dimensional pointer (hereinafter abbreviated as "distance"). However, when the size of the selection area 5050 is changed based on the distance, since it becomes more difficult to select a virtual object that is farther away from the user, it is preferable to change the size of the selection area 5050 so that the size becomes larger as the distance increases. This allows the user to easily include a virtual object that is far away in the selection area 5050. Also, the size of the selection area 5050 may be changed by the user's button operation using an operating device such as a controller, or may be changed by having the information processing device 1000 recognize the user's hand gesture. This allows the user to change the size of the selection area 5050 to a desired size.

[0044] In step S8040, the CPU 2010 (generation means) generates a subwindow (display area) that displays a list of multiple virtual objects in the selection area. Fig. 10(c) is a diagram showing a state in which virtual objects 10010, 10020, and 10030 included in the selection area 5050 in Fig. 10(a) are displayed in a list in the subwindow 10100. Therefore, when the process of step S8040 is performed, the state of Fig. 10(a) changes to the state of Fig. 10(c), and thumbnails 10110, 10120, and 10130 of the virtual objects 10010, 10020, and 10030 are displayed in a list in the subwindow 10100. This improves the visibility of the subwindow 10100.

[0045] In step S8040, as shown in FIG. 10(d), attribute information 10111, 10121, and 10131 of each virtual object 10010, 10020, and 10030 may be displayed in the sub-window 10100. In this case, when the user selects from the sub-window 10100, the user can refer to the attribute information 10111, 10121, and 10131 of each virtual object 10010, 10020, and 10030. Then, the process of FIG. 8 ends, and the process proceeds to step S7080, which will be described later. Return to the description of FIG. 7. Steps S7080 to S7100 are the same as steps S3060 to S3080 in FIG. 3, so detailed description of these steps will be omitted. Below, a case will be described in which, in the state of FIG. 10(c), the user continues to instruct the operation of selecting a virtual object. In this case, the process proceeds from step S7100 to step S7010. Furthermore, the process proceeds from step S7010 through step S7020 to step S7030, but since the selection of the virtual object has not yet been confirmed, the process proceeds to step S7050. In step S7050, since it is determined that the subwindow is being displayed in the state of Fig. 10(c), the process proceeds to the list selection operation process of step S7070.

[0046] FIG. 9 is a flowchart of the list selection operation process in step S7070 as described above. As shown in FIG. 9, in the list selection operation process, a process is performed for the operation of the user selecting a virtual object based on the position and orientation information and shape information of the virtual object in the virtual data storage unit 1030 and the position and orientation information of the three-dimensional pointer acquired in step S7020. In step S9010, the CPU 2010 determines whether or not the user has instructed to perform an operation to select a virtual object. If the CPU 2010 determines that the user has instructed to perform an operation to select a virtual object, the process proceeds to step S9020. On the other hand, if the CPU 2010 determines that the user has not instructed to perform an operation to select a virtual object, the process in FIG. 9 ends and the process proceeds to step S7080 described above. The user instructs to select a virtual object in the same manner as in the first embodiment.

[0047] In step S9020, the CPU 2010 determines whether the three-dimensional pointer is in the subwindow. This determination is made based on the position and orientation information of the three-dimensional pointer acquired in step S7020. If the CPU 2010 determines that the three-dimensional pointer is in the subwindow, the process proceeds to step S9030. On the other hand, if the CPU 2010 determines that the three-dimensional pointer is not in the subwindow, the process proceeds to step S9050. Also, if a selection instruction is given outside the subwindow, the process proceeds to step S9050, and the process proceeds to step S9050. The user may cancel the selection operation in the subwindow by operating a button on an operating device such as a controller or by having the information processing device 1000 recognize the user's hand gesture.

[0048] In step S9030, the CPU 2010 determines whether the three-dimensional pointer is pointing to a virtual object. This determination is made based on the presence or absence of interference between the three-dimensional pointer and the virtual object, based on the position and orientation information and shape information of the virtual object in the virtual data storage unit 1030 and the position and orientation information of the three-dimensional pointer acquired in step S7020. Note that the technology for determining the presence or absence of interference is a known fact, so a detailed description thereof will be omitted. The processing of FIG. 9 will be described below with reference to FIG. 10. As shown in FIG. 10(e), when the center 5020 of the three-dimensional pointer 5000 is located on the thumbnail 10120 of the virtual object 10020 in the subwindow 10100 and the virtual object 10020 is selected, the processing proceeds from step S9030 to step S9040. 10(e), if the center 5020 of the three-dimensional pointer 5000 is not located on any of the thumbnails 10110, 10120, and 10130 of the virtual objects 10010, 10020, and 10030 in the sub-window 10100, the process in Fig. 9 ends. Therefore, the process proceeds from step S9030 to the above-mentioned step S7080.

[0049] In step S9040, the CPU 2010 (confirmation means) confirms the selection of the virtual object. In the case of FIG. 10(e), as shown in FIG. 10(f), the selection of the virtual object 10020 corresponding to the thumbnail 10120 is confirmed. In this manner, the user selects the virtual object 10020. Then, the process proceeds to step S9050. In step S9050, the CPU 2010 closes the subwindow. This ends the list display in the subwindow. Then, the process of FIG. 9 ends, and the process proceeds to step S7080 described above.

[0050] As described above, according to the second embodiment, when the information processing device 1000 receives an instruction from the user to perform an operation to select a virtual object, the information processing device 1000 displays a subwindow 10100 in a mixed reality (MR) space. Furthermore, the information processing device 1000 displays a plurality of virtual objects included in a selection area 5050 near a center 5020 of a three-dimensional pointer 5000 in the subwindow 10100 in an arranged manner. Then, the user selects a desired virtual object from the subwindow 10100 with the three-dimensional pointer 5000. This allows the user to select a virtual object without accurately aligning the center 5020 of the three-dimensional pointer 5000 with the virtual object, and therefore makes it possible to easily select a virtual object far from the user or a small virtual object.

[0051] Furthermore, when a single virtual object is included in selection region 5050 near center 5020 of three-dimensional pointer 5000, information processing device 1000 confirms the selection of the single virtual object without the user selecting the virtual object from sub-window 10100. Therefore, in this case, displaying sub-window 10100 is unnecessary, but since the process of step S8040 is not performed, unnecessary display of sub-window 10100 is not performed.

[0052] The second embodiment is an example in which the present invention is applied to an MR system. However, the present invention is not limited to this, and can be applied to, for example, a VR system. That is, when the information processing device 1000 receives an instruction to perform an operation to select a virtual object from a user, the information processing device 1000 displays a sub window 10100 in a virtual reality (VR) space. In this case, the composite space image generating unit 1070 of the information processing device 1000 may display the virtual space image as it is on the image display unit 1120 of the HMD 1100 without superimposing the virtual space image on the real space image acquired by the image capturing unit 1110 of the HMD 1100. In addition, in such a VR system, when the position and orientation are measured using a sensor, the image capturing unit 1110 of the HMD 1100 may not be required.

[0053] <Other> Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0054] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) A determination means for determining whether or not a user has instructed to select a virtual object with a pointer in a mixed reality space or a virtual reality space; an enlarging means for gradually increasing a size of a selection area set in the vicinity of the pointer when the virtual object is not included in the selection area when the instruction is received, until the virtual object is included in the selection area; and determining means for determining that a virtual object included in the selection area is a virtual object selected by a user with the pointer. (Configuration 2) The information processing device according to configuration 1, wherein the instruction is given by the pointer staying within a predetermined range for a predetermined time in the mixed reality space or the virtual reality space. (Configuration 3) The information processing device according to configuration 1 or 2, characterized in that the instruction is given by a user's hand gesture in the mixed reality space or the virtual reality space, or by an operation on a user's operating device. (Configuration 4) The information processing device according to claim 2, wherein the size of the selection area increases as the time that the pointer remains within the predetermined range increases. (Configuration 5) An information processing device described in any one of configurations 1 to 4, characterized in that the size of the selection area is increased by a user's hand gesture in the mixed reality space or the virtual reality space, or by an operation on a user's operating device. (Configuration 6) A determination means for determining whether or not a user has instructed to select a virtual object with a pointer in a mixed reality space or a virtual reality space; a generating means for generating, when the instruction is given and a selection area set near the pointer includes a plurality of virtual objects, a display area in which the plurality of virtual objects included in the selection area are arranged in the mixed reality space or the virtual reality space; and determining means for determining, among a plurality of virtual objects arranged in the display area, a virtual object pointed to by a user with the pointer as the virtual object selected by the user with the pointer. (Configuration 7) The information processing device according to configuration 6, wherein the generating means arranges thumbnails of each of a plurality of virtual objects included in the selection area in the display area as a plurality of virtual objects included in the selection area. (Configuration 8) The information processing device according to configuration 6 or 7, wherein the generation means arranges attribute information of each of a plurality of virtual objects included in the selection area in the display area together with the plurality of virtual objects included in the selection area. (Configuration 9) An information processing device described in any one of configurations 6 to 8, characterized in that the instruction is performed by the pointer remaining within a predetermined range for a predetermined time in the mixed reality space or the virtual reality space. (Configuration 10) An information processing device according to any one of configurations 6 to 9, characterized in that the instruction is given by a user's hand gesture in the mixed reality space or the virtual reality space, or by operation on a user's operating device. (Configuration 11) The information processing device according to any one of configurations 6 to 10, wherein the size of the selection area increases as the distance from the starting point of the pointer to the virtual object increases. (Configuration 12) An information processing device described in any one of configurations 6 to 11, characterized in that the size of the selection area can be changed by a user's hand gesture in the mixed reality space or the virtual reality space, or by operation of the user's operating device. (Configuration 13) The information processing device according to any one of configurations 6 to 12, further comprising an enlargement means for, if the selection area does not include a virtual object, gradually increasing a size of the selection area until the selection area includes a virtual object. (Configuration 14) The information processing device according to any one of configurations 6 to 12, wherein, when the selection area includes a single virtual object, the determination means determines that the single virtual object included in the selection area is the virtual object selected by the user with the pointer. (Configuration 15) The information processing apparatus according to configuration 14, wherein the generating means does not generate the display area when the selection area includes a single virtual object. (Configuration 16) The information processing device according to any one of configurations 1 to 15, wherein the size of the selection area is defined by the distance from the center of the pointer. (Configuration 17) An information processing device according to any one of configurations 1 to 15, characterized in that the size of the selection area is defined by the angle between the line and the generating line of a cone whose axis of rotation is a line from the starting point of the pointer to the center of the pointer. (Method 1) A determination step of determining whether or not a user has instructed to select a virtual object with a pointer in a mixed reality space or a virtual reality space; an enlarging step of gradually increasing a size of a selection area set in the vicinity of the pointer when the selection area does not include a virtual object when the instruction is given, until the selection area includes a virtual object; and a determining step of determining that the virtual object included in the selection area is the virtual object selected by the user with the pointer. (Program 1) A program for causing a computer to execute each means of the information processing device according to any one of configurations 1 to 17. [Explanation of symbols]

[0055] 1000 Information processing device 2010 CPU (judging means) (enlarging means) (determining means) 5000 3D Pointer (Pointer) 5010 Virtual Objects

Claims

1. a determination means for determining whether or not a user has instructed to perform an operation of selecting a virtual object with a pointer in a mixed reality space or a virtual reality space; an enlarging means for gradually increasing a size of a selection area set in the vicinity of the pointer when the virtual object is not included in the selection area when the instruction is received, until the virtual object is included in the selection area; and determining means for determining that a virtual object included in the selection area is a virtual object selected by a user with the pointer.

2. The information processing apparatus according to claim 1 , wherein the instruction is performed by the pointer staying within a predetermined range for a predetermined time in the mixed reality space or the virtual reality space.

3. The information processing apparatus according to claim 1 , wherein the instruction is given by a user's hand gesture in the mixed reality space or the virtual reality space, or by a user's operation on an operating device.

4. 3. The information processing apparatus according to claim 2, wherein the size of the selected area increases as the time during which the pointer remains within the predetermined range increases.

5. The information processing apparatus according to claim 1 , wherein the size of the selection area is increased by a hand gesture by the user in the mixed reality space or the virtual reality space, or by an operation on a user's operating device.

6. a determination means for determining whether or not a user has instructed to perform an operation of selecting a virtual object with a pointer in a mixed reality space or a virtual reality space; a generating means for generating, when the instruction is given and a selection area set near the pointer includes a plurality of virtual objects, a display area in which the plurality of virtual objects included in the selection area are arranged in the mixed reality space or the virtual reality space; and determining means for determining, among a plurality of virtual objects arranged in the display area, a virtual object pointed to by a user with the pointer as the virtual object selected by the user with the pointer.

7. 7. The information processing apparatus according to claim 6, wherein the generating means arranges thumbnails of the respective virtual objects included in the selection area in the display area as the plurality of virtual objects included in the selection area.

8. 7. The information processing apparatus according to claim 6, wherein the generating means arranges, in the display area, attribute information of each of the plurality of virtual objects included in the selection area together with the plurality of virtual objects included in the selection area.

9. The information processing apparatus according to claim 6 , wherein the instruction is performed by the pointer staying within a predetermined range for a predetermined time in the mixed reality space or the virtual reality space.

10. The information processing apparatus according to claim 6 , wherein the instruction is given by a user's hand gesture in the mixed reality space or the virtual reality space, or by a user's operation on an operating device.

11. The information processing apparatus according to claim 6 , wherein the size of the selection area increases as the distance from the starting point of the pointer to the virtual object increases.

12. The information processing apparatus according to claim 6 , wherein the size of the selection area is changeable by a user's hand gesture in the mixed reality space or the virtual reality space, or by a user's operation on an operating device.

13. 7. The information processing apparatus according to claim 6, further comprising an enlarging means for, when the selection area does not include a virtual object, gradually increasing the size of the selection area until the selection area includes a virtual object.

14. 7. The information processing apparatus according to claim 6, wherein, when the selection area includes a single virtual object, the determination means determines that the single virtual object included in the selection area is the virtual object selected by the user with the pointer.

15. 15. The information processing apparatus according to claim 14, wherein the generating means does not generate the display area when the selection area includes a single virtual object.

16. 7. The information processing apparatus according to claim 1, wherein the size of the selected area is defined by a distance from the center of the pointer.

17. 7. The information processing apparatus according to claim 1, wherein the size of the selected area is defined by an angle between the line and a generating line of a cone having a rotation axis that is a line from the starting point of the pointer to the center of the pointer.

18. a determination step of determining whether or not a user has instructed to perform an operation of selecting a virtual object with a pointer in a mixed reality space or a virtual reality space; an enlarging step of gradually increasing a size of a selection area set in the vicinity of the pointer when the selection area does not include a virtual object when the instruction is given, until the selection area includes a virtual object; and a determining step of determining that the virtual object included in the selection area is the virtual object selected by the user with the pointer.

19. 2. A program for causing a computer to execute each of the means of the information processing apparatus according to claim 1.

Images