Information processing apparatus, information processing method and program

Abstract

To provide a technology for realizing a more accurate reflection in a virtual object.SOLUTION: An information processing apparatus that draws a first virtual object having a specular reflection component, the virtual object being arranged in a space in which a real space and a virtual space are blended, comprises: first generation means that generates a first environment map which is an image not including a first object of a plurality of objects existing in the real space, the image representing an environment of the real space, on the basis of a first imaged image imaging the real space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Mixed Reality (MR) is a technology that combines real and virtual spaces to create a space where they can interact with each other in real time. In an MR experience, the user wears a head mounted display (HMD).

[0003] When compositing real and virtual spaces, it is desirable for the optical properties of the virtual space (such as color, brightness, shadow, and reflection) to match those of the real space. The degree of match between these two optical properties is called "optical consistency."

[0004] In order to improve optical consistency, a technique for capturing light source information of a real space is known. Patent Document 1 describes a technique for capturing an image of a real space and applying the information acquired by the image capture as light source information.

[0005] As one of the techniques for capturing light source information of a real space by photographing, a technique for realizing reflection using an environment map is known. This technique is a technique for generating a two-dimensional image called an environment map by photographing a real space, and calculating reflection using the environment map. Patent Document 2 describes a technique for reducing inaccuracies in reflection by generating a first environment map representing a real space and a second environment map representing a moving object, and then using the two environment maps. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2008-33531 A [Patent Document 2] JP 2021-157458 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in Patent Document 2, in a virtual object in which reflection is realized, one subject may be reflected in two positions. For this reason, in the conventional technology, the reflection in the virtual object may be inaccurate.

[0008] Therefore, an object of the present invention is to provide a technique for realizing more accurate reflection of a virtual object. [Means for solving the problem]

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: 1. An information processing device that renders a first virtual object that is disposed in a space in which a real space and a virtual space are combined, the first virtual object having a specular reflection component, The information processing device is characterized by having a first generation means for generating a first environmental map, which is an image representing the environment of the real space based on a first captured image of the real space, and which does not include a first object among a plurality of objects existing in the real space.

[0010] One aspect of the present invention is a method for producing a composition comprising the steps of: A virtual object placed in a space that combines real space and virtual space, An information processing method for rendering a first virtual object, comprising: This information processing method is characterized by having a first generation step of generating a first environmental map, which is an image representing the environment of the real space based on a first captured image of the real space, and which does not include a first object among a plurality of objects existing in the real space. Effect of the Invention

[0011] According to the present invention, it is possible to provide a technique for realizing more accurate reflection of a virtual object. [Brief description of the drawings]

[0012] [Figure 1] 1 is a configuration diagram of a system according to a first embodiment. [Diagram 2] 1 is a hardware configuration diagram of an information processing device according to a first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating reflection according to the first embodiment. [Figure 4] 4 is a flowchart of a process for realizing reflection according to the first embodiment. [Diagram 5] FIG. 11 is a configuration diagram of a system according to a second embodiment. [Figure 6] 11 is a flowchart of a process for realizing reflection according to the second embodiment. [Figure 7] FIG. 11 is a configuration diagram of a system according to a third embodiment. [Figure 8] 13A to 13C are diagrams illustrating reflections according to a third embodiment. [Figure 9] 11 is a flowchart of a process for realizing reflection according to a third embodiment. [Figure 10] FIG. 11 is a configuration diagram of a system according to a fourth embodiment. [Figure 11] 13 is a flowchart of a process for realizing reflection according to a fourth embodiment. [Figure 12] 13A to 13C are diagrams for explaining conditions of a designated object according to the second modified example of the first embodiment. [Figure 13] FIG. 13 is a configuration diagram of a system according to a fifth embodiment. [Figure 14] 13A to 13C are diagrams illustrating the reason for sharing an environment map according to the fifth embodiment. [Figure 15] 13 is a flowchart of a process for realizing reflection according to a fifth embodiment. [Figure 16] FIG. 13 is a configuration diagram of a system according to a sixth embodiment. [Figure 17] 13 is a diagram for explaining the reason for preparing a plurality of environment maps according to the sixth embodiment. FIG. [Figure 18] 13 is a flowchart of a process for realizing reflection according to a sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the following, "reflection" in a virtual object refers to display in a virtual object having a specular reflection component (reflection performance higher than a specific value) such as a mirror or glass.

[0014] Here, it is possible to realize reflections using an environment map by approximating the reflection, but since the environment map is a two-dimensional image, it is not easy to express the three-dimensional reflection of an object. For example, since the environment map from which the reflection is calculated is a two-dimensional image and does not have a depth component, it is difficult to express the reflection taking into account the depth of the object.

[0015] One possible solution is to generate a virtual object that is a 3D reconstruction of a real object (an object that exists in real space) and perform ray tracing on the virtual object to achieve the reflection. This method makes it possible to achieve accurate reflections, but the amount of calculation required to perform a 3D reconstruction of all objects that exist in real space becomes extremely large. For this reason, it is difficult to perform real-time processing during an MR experience.

[0016] Furthermore, when two methods, a method using an environment map and a method using ray tracing, are combined, there is a possibility that a certain real object may be reflected by both methods (be reflected twice). In view of the above-mentioned problems, the following embodiments provide a technology for realizing faster and more accurate reflection (display) of a virtual object. explain.

[0017] <Embodiment 1> In the first embodiment, the system generates an environment map (a two-dimensional image of the surrounding environment that may be reflected in a virtual object) based on an image (hereinafter referred to as a "real image") of a real space (real environment) captured by an HMD 101 worn on the user's head. The system also performs three-dimensional reconstruction of a specific real object based on the real image. Then, the system realizes the reflection of the real space in the virtual object (renders the virtual object) after deleting the three-dimensionally reconstructed real object from the environment map. Hereinafter, the virtual object into which the reflection is realized is referred to as a "specular object."

[0018] 3A to 3C are diagrams for explaining the reflection of real space in a specular object. Here, the hand of a user (hereinafter, referred to as the "experiencer") wearing the HMD 101 and the real space around the experiencer are reflected in the specular object.

[0019] Fig. 3A shows a real space. Real objects 302 to 305 are objects arranged around a user 301 wearing an HMD 101. Fig. 3B shows a state of the virtual space when an environment map 306 showing the surrounding environment as shown in Fig. 3A is acquired and the hand of the user 301 is reconstructed as a virtual object 307. In the environment map 306, the real objects 302 to 305 are each represented by a two-dimensional image. A specular object 309 has a specular reflection component (reflection performance) that displays the surrounding environment by reflecting light.

[0020] With respect to the virtual space shown in Fig. 3B, reflection in specular object 309 is calculated. Fig. 3C shows the field of view of user 301 when a mixed reality image reflecting the reflection calculation result is displayed on HMD 101. As a result of realizing the reflection, "real objects 302 and 303 behind user 301 in Fig. 3A" and "user's hand 308" are reflected (rendered) in specular object 309.

[0021] 1 is a block diagram showing an example of the configuration of a system according to embodiment 1. In the system according to embodiment 1, an information processing device 100 is connected to an HMD 101 by at least one of a wired connection and a wireless connection. The information processing device 100 is a computer or the like.

[0022] The HMD 101 is a display device having a liquid crystal screen or the like. The HMD 101 is provided with a display for the right eye and a display for the left eye. The HMD 101 is attached so that each display is located in front of the right eye and the left eye of a user who wears the HMD 101 on his / her head. In addition, images having a parallax between the left and right displays are displayed in stereo on the left and right displays. In the first embodiment, the HMD 101 is an HMD (video see-through type HMD) that displays a mixed reality image based on an image of a real space captured. However, the HMD 101 may be an HMD (optical see-through type HMD) that allows a user to perceive a mixed reality space by superimposing (displaying) a virtual space image on a display medium that can be observed through the real space.

[0023] The functional configuration of the information processing device 100 will be described with reference to Fig. 1. The information processing device 100 includes an image acquisition unit 102, a measurement unit 103, an identification unit 104, a virtual object generation unit 105, a display control unit 106, a map acquisition unit 107, a deletion unit 108, a reflection calculation unit 109, a virtual data storage unit 110, a virtual image generation unit 111, and an image generation unit 112.

[0024] The image acquisition unit 102 acquires a real image (captured image) captured by an imaging unit 113 of the HMD 101 of a real space. The image acquisition unit 102 outputs the real image to the measurement unit 103, the identification unit 104, the map acquisition unit 107, the deletion unit 108, and the image generation unit 112.

[0025] The measurement unit 103 extracts characteristic information (information of points and lines, etc.) from a real image to measure the position and orientation (position and orientation) of the HMD 101. The measurement unit 103 may measure the position and orientation of the HMD 101 using infrared light, or may measure the position and orientation of the HMD 101 using a specific sensor (such as an ultrasonic sensor, a magnetic sensor, or a depth sensor).

[0026] The identification unit 104 determines whether or not a real object having a pre-specified characteristic (hereinafter, referred to as a "specified object") is included in the real image. If the specified object is included in the real image, the identification unit 104 identifies an area in the real image occupied by the specified object. In the first embodiment, the identification unit 104 determines whether or not the specified object is included in the real image according to the specified color. The identification unit 104 may determine whether or not a real object including a specific marker exists, or whether or not the specified object is included in the real image according to the shape and distance of the real object.

[0027] The virtual object generation unit 105 reconstructs the specified object in the virtual space based on the real image, the area occupied by the specified object, and information on the position and orientation of the HMD 101. In this way, the virtual object generation unit 105 generates (acquires) a three-dimensional virtual object. Then, information on the shape, color, and position of the generated virtual object is output to the display control unit 106 and the reflection calculation unit 109. In the first embodiment, the virtual object generation unit 105 performs reconstruction using stereo vision. However, the virtual object generation unit 105 may perform reconstruction by analysis using photometric stereo, or may perform reconstruction using multiple images acquired by a single camera. Hereinafter, the three-dimensional virtual object acquired by reconstruction is called a "reconstructed object."

[0028] The display control unit 106 determines whether or not to display the reconstructed object on the display unit 114. For example, the display control unit 106 determines whether or not to display the reconstructed object on the display unit 114 according to a user setting or according to the image quality of the real image. If it is determined that the reconstructed object is to be displayed, "information on the decision result as to whether or not to display the reconstructed object" (hereinafter referred to as "decision result information") and information on the reconstructed object are output to the virtual image generation unit 111. If it is determined that the reconstructed object is not to be displayed, only the decision result information is output to the virtual image generation unit 111.

[0029] The map acquisition unit 107 generates (acquires) an environmental map (a two-dimensional image showing the environment around the user) based on one or more real images. The map acquisition unit 107 can also update the current environmental map based on the real images. The technology for generating the environmental map is a known technology, so a detailed description will be omitted.

[0030] The deletion unit 108 deletes the designated object from the environmental map acquired by the map acquisition unit 107. Then, the deletion unit 108 complements the range in the environmental map from which the designated object has been deleted (hereinafter referred to as the "deletion range"). In the first embodiment, the deletion unit 108 complements the deletion range by estimating pixels in the deletion range (pixels to be placed when the designated object does not exist) based on peripheral pixels of the deletion range in the environmental map. The deletion unit 108 may complement the deletion range by estimating pixels in the deletion range based on a plurality of real images that are temporally consecutive to the real image used to generate the environmental map. The deletion range may also be complemented by processing using AI (Artificial Intelligence).

[0031] In the first embodiment, after an environment map is generated based on a real image, the designated object is deleted from the environment map, and the deleted range of the environment map is complemented. However, after the designated object is deleted from the real image, the real image may be complemented, and the environment map may be generated based on the complemented real image. In other words, the map acquisition unit 107 and the deletion unit 108 finally generate the real space. It is sufficient if an environment map that does not include a specified object existing between the two objects can be generated. Therefore, the map acquisition unit 107 and the deletion unit 108 can be collectively called a map generation unit.

[0032] Furthermore, there may be cases where the designated object is reflected in the real image even when the designated object is not reflected in the environment map acquired from the map acquisition unit 107. In such cases, the information processing device 100 may use the environment map previously acquired (generated) from the map acquisition unit 107 for drawing, without using the environment map based on the real image. In such cases, the map acquisition unit 107 may not update the current environment map based on the real image.

[0033] The reflection calculation unit 109 calculates reflection based on the position and orientation of the HMD 101, the reconstructed object, the environment map deleted and complemented by the deletion unit 108, and the virtual space data stored in the virtual data storage unit 110. Here, the reflection calculation unit 109 calculates the pixel value of each pixel of the specular object as the reflection calculation, and draws the specular object. Note that a texture (an image in which the specular object is drawn) according to the reflection calculation result is output to the virtual data storage unit 110.

[0034] Regarding the method of calculating reflection, ray tracing is performed for the reconstructed object generated by the virtual object generating unit 105 and the virtual object (virtual object representing an object that does not exist in real space) held in the virtual data holding unit 110. For the environment map, reflection calculation is performed for only one reflection on the specular object that is reflected. Note that each method is a known technique, and detailed description will be omitted (for example, see Patent Document 2).

[0035] The virtual data storage unit 110 stores data on the virtual space, such as virtual objects and light sources.

[0036] The virtual image generating unit 111 configures (generates) a virtual space based on the data of the virtual space and the determination result information. Furthermore, the virtual image generating unit 111 generates a two-dimensional image (hereinafter referred to as a "virtual image") that represents the virtual space as viewed from the position and orientation measured by the measuring unit 103. Note that the technology for generating a two-dimensional image that represents the virtual space as viewed from a predetermined position is a known technology, and therefore a detailed description thereof will be omitted.

[0037] The image generation unit 112 generates an image of a mixed reality space (a space in which a real space and a virtual space are combined) (hereinafter, referred to as a "mixed reality image") by superimposing a virtual image on a real image. The image generation unit 112 outputs the mixed reality image to the display unit 114 of the HMD 101.

[0038] Next, the functional configuration of the HMD 101 will be described with reference to Fig. 1. The HMD 101 includes an imaging unit 113 and a display unit 114.

[0039] The imaging unit 113 captures (takes) an image of a real space to obtain a real image (captured image; captured image). The imaging unit 113 outputs the real image to the information processing device 100.

[0040] The display unit 114 includes two displays. The display unit 114 displays the mixed reality image generated by the information processing device 100.

[0041] 2 is a diagram showing a hardware configuration of the information processing device 100. The information processing device 100 includes a CPU 201, a storage device 202, a ROM 203, a RAM 204, an input I / F 205, and an output I / F 206.

[0042] The CPU 201 performs overall control of each component (device) connected via the bus 200. The CPU 201 includes a processor and the like. The CPU 201 includes a ROM 203 and a RAM 204. The process steps and programs stored in 204 are executed.

[0043] The storage device 202 stores programs and data related to the information processing device 100 .

[0044] The ROM 203 stores an operating system (OS), device drivers, and a boot program.

[0045] The RAM 204 temporarily stores the programs and data loaded from the storage device 202 and the ROM 203. The RAM 204 has a work area where the CPU 201 appropriately executes each process.

[0046] The input I / F 205 is an interface through which an input signal (an input signal in a format that can be processed by the information processing device 100) is input from an external device (including the HMD 101).

[0047] The output I / F 206 is an interface that outputs an output signal in a format that can be processed by an external device (including the HMD 101) to the external device.

[0048] A process in which the information processing device 100 realizes the reflection will be described with reference to the flowchart in Fig. 4. The process in the flowchart in Fig. 4 is realized by the CPU 201 (processor or the like) of the information processing device 100 executing a program and controlling each component of the information processing device 100.

[0049] In step S400, the image acquisition unit 102 acquires a real image captured by the imaging unit 113 of the real space.

[0050] In step S401, the measurement unit 103 measures the position and orientation of the HMD 101 based on the real image acquired from the image acquisition unit .

[0051] In step S402, the map acquisition unit 107 generates (acquires) an environment map based on a real image. For example, the map acquisition unit 107 generates an environment map based on one real image, and then updates the environment map based on multiple real images, thereby generating a highly accurate environment map.

[0052] In step S403, the identification unit 104 determines (identifies) whether or not a designated object having a pre-designated characteristic is included in the real image. If it is determined that the designated object is included in the real image, the process proceeds to step S404. If it is determined that the designated object is not included in the real image, the process proceeds to step S406.

[0053] In step S404, the virtual object generation unit 105 reconstructs the designated object to generate a reconstructed object.

[0054] In step S405, the deletion unit 108 deletes the designated object from the environment map and complements the deletion range.

[0055] In step S406, the reflection calculation unit 109 calculates reflection based on the position and orientation of the HMD 101, the reconstructed object, the interpolated environment map, and the data of the virtual space. Here, the reflection calculation unit 109 renders the specular object by calculating each pixel value of the specular object as the reflection calculation. More specifically, the reflection calculation unit 109 renders the designated object on the specular object based on the reconstructed object (not based on the interpolated environment map), and renders real objects other than the designated object on the specular object based on the interpolated environment map. Note that in step S If it is determined in 403 that the specified object is not included in the real image, the reflection calculation unit 109 calculates the reflection based on the position and orientation of the HMD 101, the environment map acquired by the map acquisition unit 107, and the data of the virtual space.

[0056] According to the first embodiment, when realizing the reflection of real space in a specular object, an "environment map" and a "reconstructed object obtained by reconstructing a real object" are used in combination. This allows the reflection to be realized with higher positional accuracy than when only the environment map is used, and with less calculations than when only the reconstruction is used. Furthermore, by deleting a real object having a pre-specified characteristic from the environment map, it is possible to reduce the discomfort felt by the person experiencing the experience (user) due to the real object being reflected twice.

[0057] <Variation 1> In the first embodiment, the information processing device 100 reconstructs all designated objects having pre-specified characteristics in the virtual space. Then, the information processing device 100 realizes the reflection after removing the designated objects from the environment map. However, the information processing device 100 may not remove from the environment map and may not reconstruct an object that satisfies a specific condition among all designated objects having pre-specified characteristics. The specific condition is, for example, a condition according to the distance between the designated object and the HMD 101, or a condition according to the motion state of the real object. The specific condition may be a condition that the designated object is at a distance of a predetermined distance or more from the HMD 101, or a condition that the real object does not perform autonomous motion.

[0058] By not deleting objects that satisfy certain conditions from the environment map, the number of objects to be ray-traced is reduced, which reduces the amount of calculation required for reflection calculation.

[0059] <Variation 2> In the first embodiment, the identification unit 104 determines whether a real object including a specified color or a specific marker exists, or whether the specified object is included in the real image, depending on the shape and distance of the real object. In addition, the identification unit 104 may determine, as the specified object, a real object that satisfies the condition that it exists between the virtual object stored in the virtual data storage unit 110 and the HMD 101.

[0060] The designated object according to the second modification will be described with reference to Figs. 12A and 12B. Fig. 12A shows a mixed reality space. Real objects 1202 to 1205 are objects arranged around the user 301 wearing the HMD 101. The specular object 1206 is a virtual object, and is arranged at a position on the back side (farther away) than the real object 1205 as viewed from the HMD 101. Fig. 12B shows a state of a virtual space in which an environment map 1207 representing the surrounding environment as shown in Fig. 12A is acquired and the specular object 1206 is arranged. In the environment map 1207, the real objects 1202 to 1205 are each expressed by a two-dimensional image. The specular object 1206 has a specular reflection component (reflection performance) that displays the surrounding environment by reflecting light.

[0061] With respect to the virtual space shown in Fig. 12B, the reflection in the specular object 1206 is calculated. At this time, the real object 1205 actually exists between the HMD 101 and the specular object 1206, but in the virtual space shown in Fig. 12B, the reflection is calculated as if the real object 1205 is located farther away from the specular object 1206 with respect to the HMD 101 as a reference. Therefore, the specification unit 104 determines that the real object existing between the specular object 1206 and the HMD 101 is the designated object. In this way, the information processing device 100 prevents the real object 1205 from being reflected as a two-dimensional image in the environment map 1207, and enables the reflection to be expressed by ray tracing by reconstructing the real object 1205.

[0062] This makes it possible to remove real objects with incorrect optical consistency from the environment map, thereby achieving correct reflections.

[0063] <Embodiment 2> In the first embodiment, the information processing device generates an environment map and reconstructs a real object based on an image acquired by an imaging unit (hereinafter, referred to as an "HMD camera") mounted on the HMD. Then, the information processing device realizes a reflection based on the environment map and the reconstructed object.

[0064] In the second embodiment, the information processing device generates an environment map in advance using a real image captured by a camera (imaging device), and then reconstructs a real object.

[0065] Fig. 5 is a block diagram showing the configuration of a system in embodiment 2. In Fig. 5, the same components as those in Fig. 1 are given the same reference numerals, and the description thereof will be omitted.

[0066] The camera 500 is an imaging device having a lens, an optical system, and the like. The camera 500 outputs an image captured (captured) of the real space to the camera image acquisition unit 501. In the second embodiment, the camera 500 is an external device different from the HMD camera. Note that the camera 500 may be an HMD camera.

[0067] The information processing device 100 according to the second embodiment includes a camera image acquisition unit 501 , a map acquisition unit 502 , a physical object designation unit 503 , a physical object determination unit 504 , and a deletion unit 505 .

[0068] The camera image acquisition unit 501 acquires an image of the real space captured by the camera 500 (hereinafter, referred to as a “camera image”).

[0069] The map acquisition unit 502 generates (acquires) an environment map based on the camera image acquired by the camera image acquisition unit 501 .

[0070] The real object designation unit 503 sets conditions (hereinafter referred to as "designation conditions") for designating a real object to be deleted from the environment map and reconstructed in the virtual space. In the second embodiment, the designation conditions are conditions according to color. The designation conditions may be conditions according to shape or conditions using a marker. The designation conditions may also be a combination of these conditions.

[0071] The physical object determination unit 504 determines whether or not the environment map acquired by the map acquisition unit 502 includes a physical object that satisfies a specified condition.

[0072] If it is determined that a physical object that satisfies the specified condition is included in the environment map, the deletion unit 505 deletes the physical object from the environment map to complement the deletion range. If it is determined that a physical object that satisfies the specified condition is not included in the environment map, the deletion unit 505 does not perform any processing.

[0073] The process of realizing reflection by the information processing device 100 according to the second embodiment will be described with reference to the flowchart in FIG.

[0074] In step S600, the camera image acquisition unit 501 acquires a camera image captured by the camera 500 of the real space.

[0075] In step S601, the map acquisition unit 502 acquires the image data acquired by the camera image acquisition unit 501. An environment map is generated (acquired) based on the captured camera images.

[0076] In step S602, the physical object determination unit 504 determines whether or not the environment map includes a physical object that satisfies the designation condition designated by the physical object designation unit 503. If it is determined that the environment map includes a physical object that satisfies the designation condition, the process proceeds to step S603. If it is determined that the environment map does not include a physical object that satisfies the designation condition, the process proceeds to step S604.

[0077] In step S603, the deletion unit 108 deletes the physical object that satisfies the specified condition from the environment map acquired by the map acquisition unit 502. Then, the deletion unit 108 complements the deletion range in the environment map.

[0078] In step S604, the image acquisition unit 102 acquires a real image captured by the imaging unit 113 of the HMD 101 of the real space.

[0079] In step S605, the measurement unit 103 measures the position and orientation of the HMD 101 based on the real image acquired from the image acquisition unit 102.

[0080] In step S606, the identification unit 104 determines (identifies) whether or not a real object satisfying the specified condition is included in the real image acquired from the image acquisition unit 102. If it is determined that a real object satisfying the specified condition is included in the real image, the process proceeds to step S607. If it is determined that a real object satisfying the specified condition is not included in the real image, the process proceeds to step S608.

[0081] In step S607, the virtual object generation unit 105 reconstructs a real object that satisfies the specified condition in the virtual space to generate a reconstructed object (virtual object). At this time, the virtual object generation unit 105 may perform the reconstruction based on both the real image and the camera image, or may perform the reconstruction based on either the real image or the camera image. For example, when a real object that satisfies the specified condition is captured in the camera image, the virtual object generation unit 105 may perform the reconstruction based only on the camera image. On the other hand, when a real object that satisfies the specified condition is not captured in the camera image, the virtual object generation unit 105 may perform the reconstruction based only on the real image.

[0082] In step S608, the reflection calculation unit 109 calculates reflection based on the position and orientation information of the HMD 101, the reconstructed object, the interpolated environment map, and the virtual space data stored in the virtual data storage unit 110. Here, the reflection calculation unit 109 renders the specular object by calculating each pixel value of the specular object as the reflection calculation. If it is determined in step S606 that the designated object is not included in the real image, the reflection calculation unit 109 calculates reflection based on the environment map acquired by the map acquisition unit 502 and the virtual space data stored in the virtual data storage unit 110. That is, in this case, the reflection calculation unit 109 does not calculate the reflection of the designated object onto the specular object (does not render the designated object onto the specular object).

[0083] According to the second embodiment, an environment map based on a camera image acquired in advance can be used. As a result, the camera can capture images from any position during advance capture, so there is a low possibility that a real object will obstruct the capture during advance capture. Furthermore, advance capture allows capture over a long period of time and allows the use of a highly accurate camera, so that a highly accurate environment map can be acquired. This makes it possible to realize highly accurate reflections.

[0084] In the second embodiment, the discomfort of reflections is reduced by deleting a physical object that satisfies a pre-specified condition from the environment map when the environment map is acquired (before shooting with the HMD 101). On the other hand, after the environment map is acquired (during shooting with the HMD 101), the deletion unit 505 may delete a physical object that satisfies the specified condition from the environment map after the characteristics of the physical object are specified.

[0085] <Embodiment 3> In the third embodiment, the information processing device 100 determines whether or not the designated object is included in the environment map (camera image). When it is determined that the designated object is included in the environment map, the information processing device 100 realizes the reflection without reconstructing the designated object.

[0086] Fig. 7 is a block diagram showing the configuration of a system according to embodiment 3. The same components as those in Fig. 5 are given the same reference numerals, and their description will be omitted. The information processing device 100 has a determination unit 700 and a virtual object generation unit 701.

[0087] The determination unit 700 determines whether or not a designated object having a pre-specified characteristic is included in the environment map acquired by the map acquisition unit 502 (or the camera image acquired by the camera image acquisition unit 501).

[0088] The virtual object generation unit 701 generates a reconstructed object when it is determined that the designated object is not included in the environment map. The virtual object generation unit 701 reconstructs the designated object in a virtual space based on a real image (information on the real object) and information on the position and orientation acquired by the measurement unit 103. Then, the virtual object generation unit 701 outputs information on the reconstructed object (shape, color, position information) to the display control unit 106 and the reflection calculation unit 109.

[0089] 8A to 8D show an example of realizing the reflection of real space on a specular object in embodiment 3. In this example, the hands of a user wearing the HMD 101 and the real space around the user are reflected.

[0090] 8A shows a state of a real space in advance capturing an image for generating an environment map. The camera 500 captures an image of the surrounding environment including real objects 801-804.

[0091] FIG. 8B shows a state where a user 806 wearing the HMD 101 is experiencing MR. At this time, a real object 805 and a hand 807 of the user are specified as targets for reconstruction. In addition, the real object 803 is also specified as a target for reconstruction because it has the same characteristics as the real object 805. FIG. 8C shows the reconstructed objects and the environment map. The hand 807 of the user is generated as a reconstructed object 809. In addition, the real object 805 is generated as a reconstructed object 810. The real objects 801 to 804 are imaged as an environment map 808. FIG. 8D shows the field of view of the user after a mixed reality image in which reflection is realized is generated. In FIG. 8D, the hand 807 of the user, the real object 805, and a part of the object imaged by the environment map are reflected in the specular object 811.

[0092] The process of realizing reflection by the information processing device 100 according to the third embodiment will be described with reference to the flowchart in FIG.

[0093] In step S900, the camera image acquisition unit 501 acquires a camera image obtained by capturing a real space with the camera 500.

[0094] In step S 901 , the map acquisition unit 502 generates an environment map based on the camera image acquired by the camera image acquisition unit 501 .

[0095] In step S902, the image acquisition unit 102 acquires a real image obtained by capturing an image of the real space by the imaging unit 113.

[0096] In step S903, the measurement unit 103 measures the position and orientation of the HMD 101 based on the real image acquired by the image acquisition unit 102.

[0097] In step S904, the specification unit 104 determines (specifies) whether or not a real object (specified object) having a pre-specified characteristic is included in the real image acquired from the image acquisition unit 102. If it is determined that the specified object is included in the real image, the process proceeds to step S905. If it is determined that the specified object is not included in the real image, the process proceeds to step S907.

[0098] In step S905, the determination unit 700 determines whether or not the designated object is included in the environment map (the camera image acquired by the camera image acquisition unit 501) acquired by the map acquisition unit 502. If it is determined that the designated object is included in the environment map, the process proceeds to step S907. If it is determined that the designated object is not included in the environment map, the process proceeds to step S906.

[0099] In step S906, the virtual object generation unit 105 reconstructs the designated object to generate a reconstructed object.

[0100] In step S907, the reflection calculation unit 109 calculates reflection based on the environment map acquired by the map acquisition unit 502 and the virtual space data stored in the virtual data storage unit 110. In addition, if a reconstructed object is generated in step S906, the reflection is also calculated based on the reconstructed object. Here, the reflection calculation unit 109 renders the specular object by calculating each pixel value of the specular object as the reflection calculation. Therefore, for example, when a designated object is included in both the real image and the camera image (environment map), the reflection calculation unit 109 renders the designated object and other real objects on the specular object based on the environment map from which the designated object has not been deleted.

[0101] According to the third embodiment, among the real objects having the specified characteristics, the real objects that existed in the real space when the camera image for acquiring the environment map was captured are reflected by the environment map including the real objects. This limits the cases where ray tracing is performed, and enables the reflection to be realized at a higher speed.

[0102] <Embodiment 4> In the first to third embodiments, the realization of the reflection is described using both the environment map and the reconstructed object. In the fourth embodiment, an example is described in which the reflection is realized by acquiring a second environment map after acquiring an environment map in advance.

[0103] Fig. 10 is a block diagram showing the configuration of a system according to embodiment 4. In Fig. 10, the same components as those in Fig. 5 are given the same reference numerals, and the description thereof will be omitted. The information processing device 100 has a second map acquisition unit 1000 and a reflection calculation unit 1001.

[0104] The second map acquisition unit 1000 generates (acquires) a second environment map that includes only real objects that satisfy the specified conditions within the range of real space based on the real image, information on the position and orientation of the HMD 101, and the image area occupied by the real objects that satisfy the specified conditions.

[0105] The reflection calculation unit 1001 calculates reflection based on the environment map (hereinafter, referred to as the "first environment map") acquired from the deletion unit 505. If necessary, the second environment map is also used. The calculation method is to determine whether or not light, when reflected at a certain position of a specular object, collides with a pixel region of the second environment map occupied by the real object. If the light collides with the pixel region, the reflection calculation unit 1001 calculates the reflection for that position using the second environment map. If the light does not collides with the pixel region, the reflection calculation unit 1001 calculates the reflection for that position using the first environment map.

[0106] The process in which information processing device 100 realizes reflection will be described with reference to the flowchart in FIG.

[0107] The process of steps S1100 to S1106 is the same as the process of steps S600 to S606, and therefore description thereof will be omitted. If it is determined in step S1106 that a real object satisfying the specified condition is included in the real image, the process proceeds to step S1107. If it is determined that a real object satisfying the specified condition is not included in the real image, the process proceeds to step S1108.

[0108] In step S1107, the second map acquisition unit 1000 generates (acquires) a second environment map that represents a physical object that satisfies the specified condition.

[0109] In step S1108, the reflection calculation unit 1001 calculates reflection based on the position and orientation information measured by the measurement unit 103, the first environment map, the second environment map, and the virtual space data stored in the virtual data storage unit 110. Here, the reflection calculation unit 1001 renders the specular object by calculating each pixel value of the specular object as the reflection calculation. More specifically, the reflection calculation unit 1001 renders the real object that satisfies the specified condition on the specular object based on the second environment map, and renders the real object that does not satisfy the specified condition on the specular object based on the first environment map. Note that, if it is determined in step S1106 that the real image does not include a real object, the reflection calculation unit 1001 calculates reflection based on the first environment map and the virtual space data stored in the virtual data storage unit 110.

[0110] According to the fourth embodiment, the information processing device 100 deletes a real object that is to be reflected by the second environment map from the first environment map based on a camera image captured in advance. This makes it possible to prevent one real object from being reflected twice in the reflection. Furthermore, when the second environment map is not used, since the first environment map is a two-dimensional image, multiple real objects included in the first environment map are represented as if they exist at positions with the same sense of distance. On the other hand, by using the second environment map, a real object that satisfies a specified condition can be represented as if it exists at a position with a sense of distance different from other real objects included in the first environment map. Therefore, it is possible to represent a real object that satisfies a specified condition more three-dimensionally than when only the first environment map is used.

[0111] In addition, since the camera can be positioned freely during pre-shooting, there is little possibility that a real object will obstruct the pre-shooting. In addition, since pre-shooting allows for long-term shooting and a highly accurate camera can be used, it is possible to obtain a highly accurate environment map. Therefore, it is possible to realize highly accurate reflections.

[0112] <Embodiment 5> In the first embodiment, the information processing device 100 reconstructs all designated objects having characteristics designated in advance in a virtual space. Then, the information processing device 100 realizes the reflection after removing the designated objects from the environment map. However, in the first embodiment, the information processing device 100 reconstructs all designated objects having characteristics designated in advance in a virtual space. A process for removing the object from the environment map is required. Therefore, in the fifth embodiment, an example will be described in which, when multiple people experience a mixed reality space, the environment map is shared by multiple people, thereby reducing the process for removing the specified object from the environment map.

[0113] Fig. 13 is a block diagram showing the configuration of a system according to embodiment 5. In Fig. 13, the same components as those in Fig. 1 are given the same reference numerals, and their description will be omitted. The information processing device 100 has a map acquisition unit 1300 and a deletion unit 1301. The system also has an information processing device 1302 separate from the information processing device 100 so that the environmental map can be shared by multiple people (multiple devices).

[0114] The map acquisition unit 1300 determines whether or not there is a possibility that the deletion unit 1301 will delete a designated object from an environment map (an environment map generated by the information processing device 100). When there is a designated object to be deleted from the environment map, the map acquisition unit 1300 first acquires the environment map from the information processing device 1302. As a result, the environment map from the information processing device 1302 is shared with the information processing device 100. Since the information processing device 1302 generates an environment map different from the environment map generated by the information processing device 100, there is a possibility that the information processing device 1302 holds an environment map that does not include the designated object.

[0115] 14 shows a mixed reality space. Real objects 1403 to 1405 are objects arranged around experiencer 1401 and experiencer 1402 who are wearing the HMD 101. Virtual object 1406 is a virtual object observed in the mixed reality space by experiencer 1401 and experiencer 1402. In this case, experiencer 1402 is reflected in an environment map acquired by an information processing device corresponding to experiencer 1401. On the other hand, nothing is reflected in an environment map acquired by an information processing device corresponding to experiencer 1402. Therefore, there is no need to delete the specified object from the environment map.

[0116] We also mentioned the problem of other participants being reflected in the environment map when experiencing a multi-person experience. Regardless of the participants, it is possible that a real object is reflected in the environment map corresponding to one participant but not in the environment map corresponding to another participant. Therefore, by sharing the environment map with multiple participants, the number of processes to delete specified objects can be reduced.

[0117] The map acquisition unit 1300 not only acquires the environment map, but also sends the environment map to the information processing device 1302 when it acquires a request to acquire the environment map from the information processing device 1302. When the environment map is shared by more than two users, it may be possible to reduce the number of designated objects to be deleted by acquiring the environment map of the user who is closest to the virtual object being observed. As shown in Fig. 14, when observing one virtual object 1406, it is considered that there is a high possibility that the designated object will not be reflected in the environment map of the user who is closest to the virtual object 1406.

[0118] The deletion unit 1301 notifies the map acquisition unit 1300 of whether or not there is a possibility of deleting the designated object. In addition, the deletion unit 1301 deletes the designated object from the environment map acquired by the map acquisition unit 1300 from the information processing device 1302 or from the environment map generated by the map acquisition unit 1300.

[0119] A process in which the information processing device 100 according to the fifth embodiment realizes reflection will be described with reference to the flowchart of Fig. 15. In Fig. 15, the same processes as those in Fig. 4 are assigned the same reference numerals, and the description thereof will be omitted.

[0120] If it is determined in step S403 that the designated object is included, the process proceeds to step S1500. In step S1500, the map acquisition unit 1300 receives Obtain the environment map from the

[0121] In step S1501, the map acquisition unit 1300 compares the environment map acquired in step S1500 with the environment map acquired in step S402. If the environment map acquired in step S402 contains the designated object and the environment map acquired in step S1500 does not contain the designated object, the map acquisition unit 1300 thereafter uses the environment map acquired from the information processing device 1302. In other cases, the map acquisition unit 1300 thereafter uses the environment map acquired in step S402.

[0122] According to the fifth embodiment, the information processing device 100 can reduce the deletion process of the designated object by acquiring an environment map of an information processing device other than the information processing device 100.

[0123] <Embodiment 6> In the first embodiment, when there is one virtual object to be observed, the information processing device 100 reconstructs all designated objects having pre-specified characteristics in the virtual space. Then, the information processing device 100 realizes the reflection after removing the designated object from the environment map. In the sixth embodiment, when there are multiple virtual objects (specular objects) in the mixed reality space, an example will be described in which an environment map is prepared for each virtual object (specular object) to be observed, thereby reducing the number of designated objects to be removed and expressing a reflection with optical consistency.

[0124] Fig. 16 is a block diagram showing the configuration of a system according to embodiment 6. In Fig. 16, the same components as those in Fig. 1 are given the same reference numerals, and the description thereof will be omitted. The information processing device 100 includes a map acquisition unit 1600 and a deletion unit 1601.

[0125] First, the reason for preparing a plurality of environment maps will be described with reference to Figs. 17A to 17C. Fig. 17A shows a mixed reality space. Real objects 1702 to 1705 are objects arranged around a user 301 wearing an HMD 101. A specular object 1706 is a virtual object arranged at a position behind (farther away from) the real object 1705 as viewed from the HMD 101. A specular object 1707 is a virtual object arranged at a position in front (closer to) the real object 1705 as viewed from the HMD 101.

[0126] FIG. 17B is an environment map 1708 representing the surrounding environment as shown in FIG. 17A. When observing a specular object 1707, assuming that the specified object is an "object existing between the HMD 101 and the virtual object", the information processing device 100 reconstructs the specified object by using the environment map 1708 in FIG. 17B. This allows the user to observe a reflection with correct optical consistency without deleting the object from the environment map 1708. On the other hand, when observing a specular object 1706, the information processing device 100 deletes the specified object from the environment map 1708 in FIG. 17B to generate an environment map 1709 in FIG. 17C, and then reconstructs the real object 1705, which is the specified object. This allows the user to observe a reflection with correct optical consistency.

[0127] In this way, when the user observes the specular object 1707, the information processing device 100 only needs to save and use the environment map without deleting the designated object. On the other hand, when the user observes the specular object 1706, the information processing device 100 needs to save the environment map from which the designated object has been deleted. As described above, the designated object to be deleted differs depending on the virtual object observed by the user. For this reason, by preparing an environment map for each virtual object, it is possible to reduce the processing for deleting the designated object from the environment map and the processing for reconstructing the designated object.

[0128] The map acquisition unit 1600 generates an environment map (the surroundings of the user) based on one or more real images. When the designated object is an "object that exists between the HMD 101 and the virtual object", the map acquisition unit 1600 generates (acquires) an environment map for each virtual object.

[0129] The deletion unit 1601 deletes the designated object determined for each virtual object from the environment map for each virtual object acquired by the map acquisition unit 1600 .

[0130] Ultimately, the map acquisition unit 1600 and the deletion unit 1601 need only generate an environment map from which the designated object, which differs for each virtual object, has been deleted.

[0131] The reflection calculation unit 1602 calculates reflection based on the position and orientation of the HMD 101, the reconstructed object, the environment map for each virtual object, and the virtual object held in the virtual data holding unit 110. There is one environment map when calculating reflection. Therefore, the reflection calculation unit 1602 calculates reflection using the environment map of the virtual object that is determined to be observed by the user, among the environment maps for each virtual object. The reflection calculation unit 1602 may determine (identify) a virtual object that can be observed from the orientation (direction) of the HMD 101 as the virtual object observed by the user (hereinafter referred to as the "observed object"). If the HMD 101 has an eye tracking function that can acquire the line of sight (line of sight information) of the user, the reflection calculation unit 1602 may determine (identify) the observed object based on the eye tracking function (line of sight direction of the user). In addition, the reflection calculation unit 1602 may determine (identify) the virtual object closest to the HMD 101 as the observed object.

[0132] Here, the reflection calculation unit 1602 may use an environment map obtained by blending (combining) the environment maps of each virtual object with transparency, instead of using the environment map of the observed object, for the reflection calculation. For example, the reflection calculation unit 1602 may perform blending such that the transparency of the environment map of a virtual object that is farther away is higher, depending on the distance between the virtual object and the HMD 101. Alternatively, the reflection calculation unit 1602 may use the environment map of a specular object with the highest reflectance among the virtual objects, for the reflection calculation.

[0133] A process in which the information processing device 100 according to the sixth embodiment realizes reflection will be described with reference to the flowchart in Fig. 18. In Fig. 18, the same processes as those in Fig. 4 are denoted by the same reference numerals, and the description thereof will be omitted.

[0134] In step S1800, the map acquisition unit 1600 generates (acquires) an environment map for each virtual object based on a real image. For example, the map acquisition unit 1600 generates an environment map based on one real image, and then updates the environment map based on multiple real images. In this way, the map acquisition unit 1600 generates a highly accurate environment map.

[0135] In step S1801, the identification unit 104 determines (identifies) whether or not a "designated object having a characteristic designated in advance" for each virtual object is included in the real image. If it is determined that at least one designated object is included in the real image, the process proceeds to step S404. If it is determined that the designated object is not included in the real image, the process proceeds to step S1802.

[0136] In step S1802, the reflection calculation unit 1602 calculates reflection based on the position and orientation of the HMD 101, the reconstructed object, the environment map, and the virtual space data. Here, the reflection calculation unit 1602 calculates reflection using the environment map of the virtual object observed by the user as described above. The reflection calculation unit 1602 renders the specular object by calculating each pixel value of the specular object as the reflection calculation.

[0137] According to the sixth embodiment, the information processing device 100 can observe a reflection with correct optical consistency by acquiring an environment map for each virtual object to be observed. In addition, the deletion process of the designated object can be reduced.

[0138] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0139] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0140] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0141] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0142] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) 1. An information processing device that renders a first virtual object that is disposed in a space in which a real space and a virtual space are combined, the first virtual object having a specular reflection component, a first generation means for generating a first environment map based on a first captured image of the real space, the first environment map being an image representing an environment of the real space and not including a first object among a plurality of objects existing in the real space; 13. An information processing device comprising: (Configuration 2) 1) drawing a second object of the plurality of objects onto the first virtual object based on the first environmental map; and 2) drawing the first object onto the first virtual object without based on the first environmental map. (Configuration 3) a second generating means for generating a second virtual object which is a three-dimensional virtual object of the first object; Furthermore, the drawing means draws the first object on the first virtual object based on the second virtual object; 3. The information processing device according to configuration 2. (Configuration 4) 4. The information processing device according to configuration 3, wherein the second generating means generates the second virtual object based on the first captured image. (Configuration 5) If the first object is an object that satisfies a specific condition, the first generation means generates an environment map based on the first captured image, the environment map being an image representing an environment of the real space and including the first object; the second generating means does not generate the second virtual object, the drawing means draws the first object and the second object on the first virtual object based on the environment map generated by the first generation means; 5. The information processing device according to configuration 3 or 4. (Configuration 6) the first virtual object drawn by the drawing means is displayed on a display device worn on a user's head; The specific condition is a condition according to a distance between the display device and the first object in the real space, or a condition according to a motion state of the first object. 6. The information processing device according to configuration 5. (Configuration 7) If the first object is not included in a second captured image acquired after the first captured image, the second generating means does not generate the second virtual object, The rendering means does not render the first object on the first virtual object. 7. The information processing device according to any one of configurations 3 to 6. (Configuration 8) the second generating means generates the second virtual object based on at least one of the first captured image and the second captured image when the first object is included in a second captured image acquired after the first captured image. 8. The information processing device according to any one of configurations 3 to 7. (Configuration 9) When the first object is included in a second captured image acquired after the first captured image, and the first object is included in the first captured image, the first generation means generates an environment map based on the first captured image, the environment map being an image representing an environment of the real space and including the first object; the second generating means does not generate the second virtual object, the drawing means draws the first object and the second object on the first virtual object based on the environment map generated by the first generation means; 9. The information processing device according to any one of configurations 3 to 8. (Configuration 10) a second generation unit configured to generate a second environment map representing the first object based on a second captured image acquired after the first captured image; the drawing means draws the first object on the first virtual object based on the second environment map; 3. The information processing device according to configuration 2. (Configuration 11) the first virtual object drawn by the drawing means is displayed on a display device worn on a user's head; the first captured image is an image of the real space captured by an imaging device not included in the display device, The second captured image is an image of the real space captured by the display device. 11. The information processing device according to any one of configurations 7 to 10. (Configuration 12) the first object is an object among the plurality of objects that has a pre-specified characteristic; The second object is an object among the plurality of objects that does not have the pre-specified characteristic. 12. The information processing device according to any one of configurations 2 to 11. (Configuration 13) The pre-specified feature is a feature that exists between a display device worn on a user's head and the first virtual object. 13. The information processing device according to configuration 12. (Configuration 14) the first generating means generates the first environment map based on an image obtained by deleting the first object from the first captured image; 14. The information processing device according to any one of configurations 1 to 13. (Configuration 15) When the first object is reflected in the first captured image, the first generation means acquires the first environment map previously generated by the first generation means without using the first captured image. 14. The information processing device according to any one of configurations 1 to 13. (Configuration 16) the first generating means generates a third environment map including the first object based on the first captured image, and then generates the first environment map by deleting the first object from the third environment map; 13. The information processing device according to any one of configurations 1 to 12. (Configuration 17) when the first object is deleted from the third environmental map, the first generation means complements a first range in the third environmental map from which the first object is deleted. 17. The information processing device according to configuration 16. (Configuration 18) the first generation means estimates pixels in the first range based on neighboring pixels in the first range or a plurality of captured images that are temporally consecutive to the first captured image, thereby complementing the first range; 18. The information processing device according to configuration 17. (Configuration 19) the first generating means has a sharing means for sharing an environment map with a second information processing device different from the information processing device, the drawing means uses an environment map that does not include the first object, among a plurality of environment maps including the first environment map, for drawing the first virtual object; 19. The information processing device according to any one of configurations 2 to 18. (Configuration 20) When a plurality of the first virtual objects exist in the space obtained by fusing the real space and the virtual space, the first generation means generates an environment map for each of the first virtual objects. 19. The information processing device according to any one of configurations 2 to 18. (Configuration 21) the drawing means uses, as the first environment map, an environment map of a virtual object observed by a user, among the environment maps for each of the first virtual objects; 21. The information processing device according to configuration 20. (Configuration 22) The virtual object observed by the user is identified based on gaze information of the user. 22. The information processing device according to configuration 21. (Configuration 23) The virtual object observed by the user is identified based on the orientation of a display device worn on the user's head. 22. The information processing device according to configuration 21. (Configuration 24) the drawing means uses, for drawing, an environment map generated by adding transparency to the environment map for each of the first virtual objects and blending the environment map according to a distance between a display device worn on the user's head and the plurality of first virtual objects; 21. The information processing device according to configuration 20. (method) 1. An information processing method for rendering a first virtual object that is located in a space in which a real space and a virtual space are combined, the first virtual object having a specular reflection component, the method comprising: a first generation step of generating a first environment map based on a first captured image of the real space, the first environment map being an image representing an environment of the real space and not including a first object among a plurality of objects existing in the real space; 13. An information processing method comprising: (program) A program for causing a computer to function as each of the means of an information processing device according to any one of configurations 1 to 24. [Explanation of symbols]

[0143] 100: information processing device, 105: virtual object generation unit, 107: map acquisition unit, 108: Deletion section, 109: Reflection calculation section

Claims

1. 1. An information processing device that renders a first virtual object that is disposed in a space in which a real space and a virtual space are combined, the first virtual object having a specular reflection component, a first generation means for generating a first environment map based on a first captured image of the real space, the first environment map being an image representing an environment of the real space and not including a first object among a plurality of objects existing in the real space; 13. An information processing device comprising:

2. 2) drawing means for drawing the first object on the first virtual object without being based on the first environmental map. 3) drawing means for drawing the first object on the first virtual object without being based on the first environmental map. 4) drawing means for drawing the first object on the first virtual object without being based on the first environmental map.

3. a second generating means for generating a second virtual object which is a three-dimensional virtual object of the first object; the drawing means draws the first object on the first virtual object based on the second virtual object; 3. The information processing apparatus according to claim 2.

4. 4. The information processing apparatus according to claim 3, wherein the second generating means generates the second virtual object based on the first captured image.

5. If the first object is an object that satisfies a specific condition, the first generation means generates an environment map based on the first captured image, the environment map being an image representing an environment of the real space and including the first object; the second generating means does not generate the second virtual object, the drawing means draws the first object and the second object on the first virtual object based on the environment map generated by the first generation means; 4. The information processing apparatus according to claim 3.

6. the first virtual object drawn by the drawing means is displayed on a display device worn on a user's head; The specific condition is a condition according to a distance between the display device and the first object in the real space, or a condition according to a motion state of the first object.

6. The information processing apparatus according to claim 5,

7. If the first object is not included in a second captured image acquired after the first captured image, the second generating means does not generate the second virtual object, the rendering means does not render the first object on the first virtual object; 4. The information processing apparatus according to claim 3.

8. the second generating means generates the second virtual object based on at least one of the first captured image and the second captured image when the first object is included in a second captured image acquired after the first captured image.

4. The information processing apparatus according to claim 3.

9. When the first object is included in a second photographed image acquired after the first photographed image, and the first object is included in the first photographed image, the first generation means generates an environment map based on the first captured image, the environment map being an image representing an environment of the real space and including the first object; the second generating means does not generate the second virtual object, the drawing means draws the first object and the second object on the first virtual object based on the environment map generated by the first generation means; 4. The information processing apparatus according to claim 3.

10. a second generation unit configured to generate a second environment map representing the first object based on a second captured image acquired after the first captured image; the drawing means draws the first object on the first virtual object based on the second environment map; 3. The information processing apparatus according to claim 2.

11. the first virtual object drawn by the drawing means is displayed on a display device worn on a user's head; the first captured image is an image of the real space captured by an imaging device not included in the display device, The second captured image is an image of the real space captured by the display device.

11. The information processing apparatus according to claim 7,

12. the first object is an object among the plurality of objects that has a pre-specified characteristic; the second object is an object among the plurality of objects that does not have the pre-specified characteristic; 11. The information processing apparatus according to claim 2,

13. The pre-specified feature is a feature that exists between a display device worn on a user's head and the first virtual object.

13. The information processing apparatus according to claim 12.

14. the first generating means generates the first environment map based on an image obtained by deleting the first object from the first captured image; 11. The information processing apparatus according to claim 1,

15. When the first object is reflected in the first photographed image, the first generation means acquires the first environment map previously generated by the first generation means without using the first photographed image.

11. The information processing apparatus according to claim 1,

16. the first generating means generates a third environment map including the first object based on the first captured image, and then generates the first environment map by deleting the first object from the third environment map; 11. The information processing apparatus according to claim 1,

17. when the first object is deleted from the third environmental map, the first generation means complements a first range in the third environmental map from which the first object is deleted.

17. The information processing apparatus according to claim 16,

18. the first generation means estimates pixels in the first range based on peripheral pixels in the first range or a plurality of captured images that are temporally consecutive to the first captured image, thereby complementing the first range; 18. The information processing apparatus according to claim 17,

19. the first generating means has a sharing means for sharing an environment map with a second information processing device different from the information processing device, the drawing means uses an environment map that does not include the first object, among a plurality of environment maps including the first environment map, for drawing the first virtual object; 11. The information processing apparatus according to claim 2,

20. when a plurality of the first virtual objects exist in the space obtained by fusing the real space and the virtual space, the first generation means generates an environment map for each of the first virtual objects.

11. The information processing apparatus according to claim 2,

21. the drawing means uses, as the first environment map, an environment map of a virtual object observed by a user, among the environment maps for each of the first virtual objects; 21. The information processing apparatus according to claim 20,

22. The virtual object observed by the user is identified based on gaze information of the user.

22. The information processing apparatus according to claim 21 .

23. The virtual object observed by the user is identified based on the orientation of a display device worn on the user's head.

22. The information processing apparatus according to claim 21 .

24. the drawing means uses, for drawing, an environment map generated by adding transparency to the environment map for each of the first virtual objects and blending the environment map in accordance with a distance between a display device worn on a user's head and the plurality of first virtual objects; 21. The information processing apparatus according to claim 20,

25. 1. An information processing method for rendering a first virtual object that is located in a space in which a real space and a virtual space are combined, the first virtual object having a specular reflection component, the method comprising: a first generation step of generating a first environment map based on a first captured image of the real space, the first environment map being an image representing an environment of the real space and not including a first object among a plurality of objects existing in the real space; 13. An information processing method comprising:

26. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 10.

Images