Enhanced specular reflections for inset content

Abstract

Systems and methods for enhancing specular reflections are provided. Exemplary methods may include determining a first portion of specular reflections associated with a computer-generated object based on a first contribution of an environment map component at a shading point from the computer-generated object, and determining a second portion of the specular reflections associated with the computer-generated object based on a second contribution from a camera feed component at an intersection of the camera feed and a reflection vector associated with the environment map component. The exemplary method may also include determining the specular reflections associated with the computer-generated object at the shading point based on a blend of the first portion and the second portion of the specular reflections.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of and claims priority to U.S. Application No. 16 / 374,406, filed on April 3, 2019, which claims the benefit of U.S. Provisional Patent Application No. 62 / 653,889, filed on April 6, 2018, both disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to enhanced specular reflection for inserting content. Background Art

[0004] Content can be inserted into an image or a user's field of view. For example, an augmented reality (AR) system can generate an immersive augmented environment for a user by inserting content. An immersive augmented environment can be generated by superimposing computer-generated content on the user's field of view of the real world. For example, computer-generated content can include three-dimensional entities, labels, text information, images, sprites, etc. These images can be displayed at a certain position in the user's field of view so as to appear to cover objects in the real world. Similarly, computer-generated content can be overlaid on a displayed image. The lighting applied to the inserted content may affect the degree of fit between the inserted content and the rest of the image. Summary of the invention

[0005] The present disclosure describes systems and methods for enhanced specular reflections for inserting content. In one embodiment, the exemplary method may include determining a first portion of specular reflections associated with a computer-generated object based on a first contribution from an environment map component at a shading point of the computer-generated object; and determining a second portion of the specular reflections associated with the computer-generated object based on a second contribution from a camera feed component at an intersection of the camera feed and a reflection vector associated with the environment map component. The exemplary method may also include determining the specular reflection associated with the computer-generated object at the shading point based on a blend of the first portion and the second portion of the specular reflection.

[0006] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are included to provide a further understanding of the subject technology and are incorporated in and constitute a part of this specification, illustrate disclosed aspects of the subject technology and together with the description serve to explain the principles of the subject technology.

[0008] Figure 1 is a block diagram illustrating a system according to an exemplary embodiment.

[0009] Figure 2 is a third-person perspective of an exemplary physical space, where the user is moving through Figure 1 An exemplary HMD for experiencing an AR environment.

[0010] Figure 3A , Figure 3B and Figure 3C is a diagram showing an exemplary head mounted display device and controller according to an implementation as described herein.

[0011] Figure 4 is a schematic diagram of a user experiencing an AR environment via an exemplary portable electronic device.

[0012] Figure 5 is an illustration of an exemplary implementation of enhancing specular reflection according to implementations described herein.

[0013] Figure 6 A flow chart is shown of an exemplary method of determining specular reflection according to implementations described herein.

[0014] Figure 7 Examples of computer devices and mobile computer devices that can be used to implement the techniques described herein are shown. DETAILED DESCRIPTION

[0015] Reference will now be made in detail to non-limiting examples of the present disclosure, examples of which are shown in the accompanying drawings. The examples are described below with reference to the accompanying drawings, wherein like reference numerals represent like elements. When like reference numerals are shown, the corresponding description is not repeated, and the interested reader is referred to the previously discussed drawings for a description of the same elements.

[0016] The augmented reality (AR) system includes a system for inserting computer-generated (CG) content into a user's perception of a physical space around the user. The computer-generated content may include three-dimensional entities, labels, text information, images, sprites, three-dimensional entities, etc. In some embodiments, the inserted content is for entertainment, educational, or informational purposes.

[0017] An exemplary AR system is a portable electronic device, such as a smart phone, including a camera and a display device. The portable electronic device can capture images using a camera and show an AR image on the display device, the AR image including computer-generated content overlaid on the image captured by the camera. Another exemplary AR system includes a head-mounted display (HMD) worn by a user. The HMD includes a display device positioned in front of the user's eyes. For example, the HMD may cover the user's entire field of view so that the user can only see the content displayed by the display device. In some examples, the display device is configured to display two different images, one of which can be seen by each eye of the user. For example, at least some of the content in one of the images can be slightly offset relative to the same content in the other image so as to generate a perception of a three-dimensional scene due to parallax. In some embodiments, the HMD includes a chamber in which a portable electronic device such as a smart phone can be placed to allow the display device of the portable electronic device to be viewed through the HMD.

[0018] Additional exemplary AR systems include an HMD that allows a user to see a physical space while wearing the HMD. The HMD may include a microdisplay that displays computer-generated content overlaid on the user's field of view. For example, the HMD may include an at least partially transparent shield that includes a combiner that allows light from the physical space to reach the user's eyes while reflecting an image displayed by the microdisplay toward the user's eyes.

[0019] When computer-generated content is inserted into an image, lighting can be applied to the content so that the content more closely matches the rest of the image. For example, a three-dimensional model can be rendered using a light source or an ambient lighting map that approximates the lighting conditions captured in the image. An AR system may need to refresh the image displayed to the user in real time at a high rate (e.g., 24 frames per second (FPS), 30 FPS, 60 FPS, or other rates). Traditional techniques for determining or estimating the lighting in a scene based on an image may require so many calculations (or processor cycles) that they cannot be performed in an AR system. Some traditional techniques require prior information about the lighting in the environment, which may not be available for many of the environments in which the AR system is used.

[0020] Techniques described herein incorporate details from a camera feed into specular reflections to improve the realism of inserted content (e.g., computer-generated (CG) objects). In other words, the techniques describe incorporating details from a camera feed of a device to improve the appearance of an inserted object. In one embodiment, the improvement or enhancement can be achieved by using a camera feed (e.g., an image, a video, etc.) acquired by a camera of the device and incorporating the camera feed (e.g., at least a portion of the camera feed) into specular reflections that are rendered on the inserted CG object to enhance the appearance of the rendering of surfaces commonly found in nature.

[0021] Specular reflection can be defined as a type of surface reflection where the angle of reflection is equal to the angle of incidence, and where the incident light is (mostly) reflected into a single outgoing direction. Specular reflection occurs on most smooth and polished surfaces (e.g., mirrors, glossy surfaces, and metallic surfaces) when light is reflected from a surface where there is no absorption and no significant scattering, and typically reveals visual details of the reflected area from space. In order to convincingly render shiny computer-generated objects inserted into a camera feed (e.g., an image feed or a video feed), specular reflections on the surface of the computer-generated object can be collected, and details from the camera feed can be incorporated into the specular reflections to significantly improve the realism of the inserted computer-generated object. As described in detail below, this can enhance the user's immersion and experience.

[0022] Accurate rendering of specular reflections requires knowledge of incident light from all directions in the surrounding environment at a high angular resolution. Because a typical camera on a device only covers a narrow solid angle (e.g., 75°×35°), all necessary or desired information (e.g., information on which the reflected radiance is based) may not be available. However, the information available from the camera feed can be mixed with a synthetic representation of the lighting environment (e.g., an environment map) to improve realism that would not be possible without the techniques described in the present disclosure. For example, the specular reflection can be split or divided into two components (e.g., at least two components), namely an environment map component and a camera feed component. The two components are then mixed (e.g., combined) based on a function of the angle between a negative view vector (e.g., a vector in a direction opposite to the view vector) and a surface normal vector. For example, in one embodiment, specular reflection for enhancing user experience can be based on the following steps: determining an environment map component at a shading point of a computer-generated object; determining a camera feed component at the intersection of a reflection vector associated with the environment map component and a camera feed; and determining a specular reflection (or enhanced specular reflection) at the shading point based on a blend of the environment map component and the camera feed component.

[0023] Figure 11 is a block diagram illustrating a system 100 according to an exemplary embodiment. The system 100 generates an augmented reality (AR) environment to a user of the system 100. In some embodiments, the system 100 includes a computing device 102, a head mounted display device (HMD) 104, and an AR content source 106. A network 108 is also shown, through which the computing device 102 can communicate with the AR content source 106.

[0024] The computing device 102 may include a memory 110, a processor component 112, a communication module 114, a sensor system 116, and a display device 118. The memory 110 may include an AR application 120, an AR content 122, an image buffer 124, an image analyzer 126, a lighting engine 128, and a rendering engine 130. The computing device 102 may also include various user input components (not shown), such as a controller that communicates with the computing device 102 using a wireless communication protocol. In some embodiments, the computing device 102 may be a mobile device (e.g., a smart phone) configured to provide or output AR content to a user via the HMD 104. For example, the computing device 102 and the HMD 104 may communicate via a wired connection (e.g., a universal serial bus (USB) cable) or via a wireless communication protocol (e.g., Wi-Fi, Bluetooth, Zigbee, etc.). Additionally or alternatively, the computing device 102 may be a component of the HMD 104 and may be contained within a housing of the HMD 104.

[0025] The memory 110 may include one or more non-transitory computer-readable storage media. The memory 110 may store instructions and data that may be used to generate an AR environment for a user.

[0026] The processor component 112 includes one or more devices capable of executing instructions to perform various tasks associated with generating an AR environment, such as instructions stored by the memory 110. For example, the processor component 112 may include a central processing unit (CPU) and / or a graphics processor unit (GPU). In some embodiments, if a GPU is present, some image / video rendering tasks (such as shading content based on determined lighting parameters) may be offloaded to the GPU.

[0027] The communication module 114 includes one or more devices for communicating with other computing devices, such as the AR content source 106. The communication module 114 may communicate via a wireless or wired network, such as the network 108.

[0028] The sensor system 116 may include various sensors, such as a camera assembly 132. Embodiments of the sensor system 116 may also include other sensors, including, for example, an inertial motion unit (IMU) 134, light sensors, audio sensors, image sensors, distance and / or proximity sensors, contact sensors such as capacitive sensors, timers and / or other sensors, and / or different combinations of sensors.

[0029] The IMU 134 detects motion, movement, and / or acceleration of the computing device 102 and / or the HMD 104. The IMU 134 may include a variety of different types of sensors, such as accelerometers, gyroscopes, magnetometers, and other such sensors. The position and orientation of the HMD 104 may be detected and tracked based on data provided by the sensors included in the IMU 134. The detected position and orientation of the HMD 104 may allow the system to further detect and track the user's gaze direction and head movements.

[0030] In some implementations, the AR application may use the sensor system 116 to determine the user's position and orientation within the physical space, and / or to identify features or objects within the physical space.

[0031] The camera assembly 132 captures images and / or video of the physical space surrounding the computing device 102. The camera assembly 132 may include one or more cameras. The camera assembly 132 may also include an infrared camera.

[0032] The AR application 120 may present or provide AR content to the user via one or more output devices (such as a display device 118, a speaker, and / or other output devices) of the HMD and / or computing device 102. In some embodiments, the AR application 120 includes instructions stored in the memory 110, which, when executed by the processor component 112, cause the processor component 112 to perform the operations described herein. For example, the AR application 120 may generate an AR environment based on, for example, AR content (such as AR content 122 and / or AR content received from the AR content source 106) and present it to the user. The AR content 122 may include content such as images or videos, which may be displayed on a portion of the user's field of view in the HMD 104. For example, the content may be generated using lighting that substantially matches the physical space in which the user is located. The content may include objects for covering various portions of the physical space. The content may be rendered as a flat image or a three-dimensional (3D) object. A 3D object may include one or more objects represented as a polygonal mesh. The polygonal mesh may be associated with various surface textures (such as colors and images). The polygonal mesh may be shaded based on various lighting parameters.

[0033] The AR application 120 can use the image buffer 124, the image analyzer 126, the lighting engine 128, and / or the rendering engine 130 to generate an image for display via the HMD 104 based on the AR content 122. For example, the AR application 120 can use the image buffer 124, the image analyzer 126, the lighting engine 128, and / or the rendering engine 130 to determine an environment map component at a shading point of a computer-generated object, determine a camera feed component at an intersection of a reflection vector associated with the environment map component and the camera feed, and determine a specular reflection at the shading point based on a blend of the environment map component and the camera feed component.

[0034] In some embodiments, the image buffer 124 is an area of ​​the memory 110 configured to store one or more images. In some embodiments, the computing device 102 stores images acquired by the camera component 132 as textures within the image buffer 124. Alternatively or in addition, the image buffer may also include a memory location integral to the processor component 112, such as a dedicated random access memory (RAM) on a GPU.

[0035] In some implementations, the image analyzer 126, the lighting engine 128, and the rendering engine 130 may include instructions stored in the memory 110 that, when executed by the processor component 112, cause the processor component 112 to perform the operations described herein to generate an image or series of images for display to a user (e.g., via the HMD 104).

[0036] The AR application 120 may update the AR environment based on input received from the camera assembly 132, the IMU 134, and / or other components of the sensor system 116. The position and orientation of the HMD 104 may be detected and tracked based on data provided by sensors included in the IMU 134. The detected position and orientation of the HMD 104 may allow the system to further detect and track the position and orientation of the user within the physical space. Based on the detected position and orientation, the AR application 120 may update the AR environment to reflect the user's changed orientation and / or position within the environment.

[0037] Although in Figure 1 104 are shown as separate devices, but in some embodiments, the computing device 102 can include the HMD 104. In some embodiments, the computing device 102 communicates with the HMD 104 via a cable, such as Figure 1 For example, computing device 102 may transmit video signals and / or audio signals to HMD 104 for display to a user, and HMD 104 may transmit motion, position, and / or orientation information to computing device 102.

[0038] The AR content source 106 can generate and output AR content, which can be distributed or sent to one or more computing devices, such as computing device 102, via network 108. In an exemplary embodiment, the AR content includes a three-dimensional scene and / or an image. In addition, the AR content may include an audio signal / video signal that is streamed or distributed to one or more computing devices. The AR content may also include an AR application running on computing device 102 to generate a 3D scene, an audio signal, and / or a video signal.

[0039] The network 108 may be the Internet, a local area network (LAN), a wireless local area network (WLAN), and / or any other network. For example, the computing device 102 may receive an audio signal / video signal via the network, and in an illustrative exemplary embodiment, the audio signal / video signal may be provided as part of the AR content.

[0040] Figure 2 2 is a third-person perspective of an exemplary physical space 200 in which a user is experiencing an AR environment 202 through an exemplary HMD 104. The AR environment 202 is generated by an AR application 120 of a computing device 102 and displayed to the user through the HMD 104.

[0041] The AR environment 202 includes inserted content 204 displayed on the image of the physical space 200. In an exemplary embodiment, the content 204 can be a sphere inserted into the AR environment 202. In such an exemplary embodiment, a subtle reflection along the silhouette edge of the shiny surface of the sphere driven by the camera feed can be introduced to enhance the specular reflection. This works very well because the reflection vectors are not too far away from the view vectors, and thus allow them to be reasonably approximated based on the visible pixels of the camera feed and blended with the synthetic environment map.

[0042] In some embodiments, the AR environment 202 is provided to the user as a single image or a pair of stereo images that substantially occupies all of the user's field of view and is displayed to the user via the HMD 104. In other embodiments, the AR environment is provided to the user by displaying / projecting the inserted content 204 on an at least partially transparent combiner that occupies at least a portion of the user's field of view. For example, portions of the HMD 104 may be transparent, and while wearing the HMD 104, the user may be able to see the physical space 200 through these portions.

[0043] Figure 3A and Figure 3B is an exemplary HMD 300 (such as Figure 2104) worn by a user in FIG. Figure 3C An exemplary handheld electronic device 302 for controlling and / or interacting with the HMD 300 is shown.

[0044] The handheld electronic device 302 may include a housing 303 in which the internal components of the device 302 are stored and a user interface 304 accessible to a user outside the housing 303. The user interface 304 may include a touch-sensitive surface 306 configured to receive a user touch input. The user interface 304 may also include other components for manipulation by the user, such as actuation buttons, knobs, joysticks, etc. In some embodiments, at least a portion of the user interface 304 may be configured as a touch screen, wherein the portion of the user interface 304 is configured to display user interface items to the user and is also configured to receive touch input from the user on the touch-sensitive surface 306. The handheld electronic device 302 may also include a light source 308, which is configured to selectively emit light, such as a beam or light, through a port in the housing 303, for example, in response to receiving a user input at the user interface 304.

[0045] The HMD 300 may include a housing 310 coupled to a frame 320, to which an audio output device 330 including, for example, a speaker mounted in an earphone is also coupled. Figure 3B 310b of the housing 310 so that some of the components received in the housing 310 are visible. The display 340 can be mounted on the inwardly facing side of the front portion 310a of the housing 310. When the front portion 310a is in a closed position relative to the base 310b of the housing 310, a lens 350 can be mounted in the housing 310 between the user's eyes and the display 340. In some embodiments, the HMD 300 can include a sensing system 360 and a control system 370, the sensing system 360 including various sensors, and the control system 370 including a processor 390 and various control system devices to facilitate the operation of the HMD 300.

[0046] In some embodiments, the HMD 300 may include a camera 380 to capture still and moving images. The images captured by the camera 380 may be used to help track the physical location of the user and / or the handheld electronic device 302 in the real world or in physical space relative to the augmented environment, and / or may be displayed to the user in a pass-through mode on the display 340 to allow the user to temporarily leave the augmented environment and return to the physical environment without having to remove the HMD 300 or otherwise change the configuration of the HMD 300 to move the housing 310 out of the user's line of sight.

[0047] For example, in some embodiments, the sensing system 360 may include an inertial measurement unit (IMU) 362 that includes various different types of sensors, such as accelerometers, gyroscopes, magnetometers, and other such sensors. The position and orientation of the HMD 300 may be detected and tracked based on data provided by the sensors included in the IMU 362. The detected position and orientation of the HMD 300 may allow the system to further detect and track the user's head gaze direction and movement.

[0048] In some embodiments, the HMD 300 may include a gaze tracking device 365 to detect and track the user's eye gaze. The gaze tracking device 365 may include, for example, an image sensor 365A or multiple image sensors 365A to capture images of the user's eyes, such as a specific part of the user's eyes (such as the pupil), to detect and track the direction and movement of the user's gaze. In some embodiments, the HMD 300 may be configured so that the detected gaze is processed as a user input to be converted into a corresponding interaction in an immersive virtual experience.

[0049] In some embodiments, HMD 300 includes a portable electronic device, such as a smartphone, that is removably disposed within a chamber of housing 310. For example, display 340 and camera 380 may be provided by the portable electronic device. Figure 3A ), display 340 is aligned with lens 350 so that the user can view at least a portion of display 340 (provided by the portable electronic device) through each eye. Camera 380 can be aligned with a hole in housing 310 so that the portable electronic device of HMD 300 can capture images when placed in housing 310.

[0050] Figure 4 4 is a schematic diagram 400 of a user experiencing an AR environment 202 via an exemplary portable electronic device 402. Portable electronic device 402 is an example of computing device 102. Portable electronic device 402 may be a smart phone, tablet computer, or other type of portable computing device. In this example, the user experiences the AR environment through a display device 418 of the portable electronic device. For example, display device 418 may include a screen that may show images and / or videos.

[0051] Figure 5 An exemplary embodiment 500 for determining the specular reflection of inserted content is shown. In one exemplary embodiment, it may be possible to Figure 2 The specular reflection (or enhanced specular reflection) of the same or similar inserted content 204 (eg, the inserted computer-generated object 504 ) may be determined at the computing device.

[0052] In an exemplary embodiment, Figure 5 The camera 502, computer-generated object 504, environment map component (Ce) 506, camera feed component (Cc) 508, shading point 510, reflection vector 512, view vector 514, surface normal vector 516, intersection point 518, and / or negative view vector 524 (opposite to view vector 514) may be included. Angle θ 520 represents a measure of the angle between negative view vector 524 and surface normal vector 516, while angle Φ 522 represents a measure of the angle between view vector 514 and reflection vector 512. In some embodiments, for example, specular reflections associated with computer-generated object 504 may be determined based on a blend of contributions from environment map components and camera feed components (e.g., environment map component 506 and camera feed component 508).

[0053] For example, a contribution (e.g., a first contribution) from an environment map component 506 may be determined from an environment map represented, for example, by a giant sphere surrounding the camera 502. The environment map component 506 may be determined from the environment map using any environment mapping technique. An environment mapping technique may be an image-based lighting technique for approximating the appearance of the reflective surface by using a texture image (e.g., a pre-computed texture image). In an exemplary embodiment, a cube map may be used to determine the environment map component 506. For example, the environment may be projected onto the sides of a cube and stored as six textures or unwrapped into six regions of a single texture.

[0054] In some embodiments, the contribution from the environment map component 506 can be a synthetic map (e.g., a cube map) of a fixed scene. The map can be created (e.g., synthesized) by a high dynamic range acquisition, which can be convolved (e.g., blurred) using a filter (e.g., a cosine lobe filter) to simulate a predefined specular reflectivity lobe, for example. Pre-convolution can produce physically accurate specular highlights for surfaces with very low roughness. In some embodiments, to handle surfaces with higher roughness, the environment can be blurred by, for example, a specific amount using a filter (e.g., a mipmap filter). Cosine lobe pre-convolution and mipmap-based blurring techniques are a fast and / or convenient way to calculate synthetic specular reflections from an image-based light source (IBL). In some embodiments, the blurring can be applied to different degrees at rendering time. For example, the blurring can be applied by changing the mipmap detail level (LOD) based on a measure of specular roughness (e.g., this can be performed in a similar manner to specular reflections from a synthetic cube map). In some embodiments, the method can provide an inexpensive approximation to the correct surface bidirectional reflectance distribution function (BRDF).

[0055] In an exemplary embodiment, the camera 502 may acquire various frames of images (referred to as camera feeds), and the pixel color at the intersection point 518 (referred to as the camera feed component 508) may represent a color that may be used to blend with the environment map component 506 to generate a specular reflection at the shadow processing point 510. The intersection point 518 may be the intersection of the reflection vector 512 and the camera feed. For example, the intersection point 518 may be the intersection of the reflection vector 512 and the camera feed plane, which may be, for example, a telescopic plane facing the camera 502 and sized to fill the camera's field of view, onto which the camera feed image is projected. In an exemplary embodiment, the reflection vector 512 may originate from the shadow processing point 510 and may extend in the environment map component 506 and may pass through the camera feed. The camera 504 may be at the shadow processing point 510 or any other point. In some embodiments, as described above, the camera feed may be blurred based on surface roughness to simulate specular lobes consistent with the specular lobes of the cube map. In other words, the environment map and the camera feed may be blurred in a consistent manner to achieve a correct blend between the two components. However, in some embodiments, there may be some subtle differences between the blurring of the camera feed and the blurring of the cube map. For example, pre-convolution may be skipped when blurring the camera feed.

[0056] The specular reflection at the shading point 510 can be determined in several ways. In some embodiments, the specular reflection can be determined based on a blend of an environment map component and a camera feed component. In an exemplary embodiment, the specular reflection can be determined by calculating the dot product of a reflection vector and a view vector (e.g., the dot product of reflection vector 512 and view vector 514). In some embodiments, before calculating the dot product for determining the specular reflection, the vectors (e.g., the reflection vector and the view vector) can be normalized to a length of 1 to produce, for example, the cosine of the angle between the two vectors, clamped at a minimum value of 0, and raised to the fifth power, as shown in equation (1) below. However, in some embodiments, the specular reflection can be determined by calculating the dot product of the reflection vector and the view vector 514. Indirectly calculate the value of cosθ, where is the dot product between the unit view vector and the unit reflection vector (e.g., based on normalization to length 1), and

[0057] b = (1 – cos θ) 5 …………(1)

[0058] C f = (1-b) C c + bC e …… (2)

[0059] In the above equation, C f represents specular reflection, C c represents the camera feed component, and C e Denotes the environment map component. The above equations (1) and (2) are closely related to the estimate of the Fresnel reflectance (also known as the Schilck approximation) and can be used to determine specular reflections, since the reflectivity of shiny surfaces is very view-dependent.

[0060] Figure 6 6 is a flow chart of an exemplary method 600 for determining specular reflection. In an exemplary embodiment, the method 600 may be performed by the computing device 102 to provide, for example, a better AR experience to a user.

[0061] At block 610, the computing device may determine a portion of the specular reflection associated with the computer-generated object. For example, in one embodiment, the computing device 102 may determine a first portion of the specular reflection associated with the computer-generated object (e.g., the computer-generated object 504). In some embodiments, the first portion may be determined based on a contribution (e.g., a first contribution) from an environment map component (e.g., the environment map component 506). The environment map component 506 may be an environment map component at a shadow processing point (e.g., the shadow processing point 510) and may be determined from a scene (e.g., an image, a video, etc.) of a surrounding environment. In some embodiments, the environment map component 506 may be determined using an environment mapping technique. An environment mapping technique may generally be defined as approximating the appearance of a reflective surface by using a texture image (e.g., a pre-computed texture image). In an exemplary embodiment, a cube map may be used to determine the environment map component 506 at the shadow processing point 510.

[0062] At block 620, the computing device may determine a second portion of the specular reflection associated with the computer-generated object. For example, in one embodiment, the computing device 102 may determine a second portion of the specular reflection associated with the computer-generated object (e.g., computer-generated object 504). In some embodiments, the second portion may be determined based on a contribution (e.g., a second contribution) of a camera feed component (e.g., camera feed component 508) from a camera feed at an intersection point (e.g., intersection point 518). The intersection point 518 may be an intersection point of the camera feed and a reflection vector (e.g., reflection vector 512). In an exemplary embodiment, the reflection vector 518 may be associated with the environment map component 506.

[0063] At block 630, the computing device may determine the specular reflection at the shadow processing point based on the blending of the first portion and the second portion of the specular reflection. For example, in one embodiment, the computing device 102 may determine the specular reflection at the shadow processing point (e.g., the shadow processing point 510) based on the blending of the first portion of the specular reflection and the second portion of the specular reflection (e.g., the first portion and the second portion of the specular reflection as determined above at blocks 610 and 620).

[0064] In some embodiments, the blending of the first and second portions can be based on an angle between a negative view vector (e.g., 524) and a surface normal vector (e.g., 516). For example, based on equations (1) and (2) above, reflectivity can become significantly stronger at grazing angles (e.g., a light beam that is nearly parallel to the surface of the computer-generated object 504) than at direct view. In an exemplary embodiment, for example, equation (2) for determining the specular reflection based on a blend between the environment map component 506 and the camera feed component 508 can include more of the camera feed (e.g., biased toward the camera feed component 508) because the reflection vector 512 tends to fall within the boundaries of the available camera image, such as within a narrow cone in front of the camera 502. In an exemplary embodiment, when the reflection vector 512 may be significantly to the side or behind the camera 502, the specular reflection can be determined by blending the environment map component 506 and the camera feed component 508 to favor (e.g., bias toward) the environment map component 506 because the environment map component 506 can provide composite information about all directions.

[0065] Different computer-generated objects reflect light differently. For example, for smooth / shiny objects, a perfect mirror image is usually received. The color of the pixel of the camera feed can be used to blend with the environment map component 506. However, for some mirror surfaces that are not so smooth, such as rougher surfaces (such as brushed aluminum, non-shiny plastic, etc.), light may be scattered. In some embodiments, the image from the camera feed can be blurred in the reflection of the mipmap filter. In other words, instead of using the color of a specific pixel in the camera feed, the average color of multiple pixels within a certain box centered on the intersection is used. In other words, as described above, different specular lobes can be simulated for the environment map component 506 and the camera feed component 508 by blurring the corresponding textures. In some embodiments, the blurring process can be applied to different degrees at rendering time. This can be effectively achieved by changing the mipmap detail level (LOD) based on a measure of the mirror roughness (performed in a similar manner to the mirror reflection from the synthetic cube map). This provides an inexpensive approximation to the correct surface bidirectional reflectance distribution function (BRDF). In some implementations, blurring (eg, consistent blurring) of two textures (eg, an environment map component and a camera feed component) can produce a plausible blend to enhance the specular reflections of the inserted content.

[0066] In some implementations, the camera feed texture may be applied to an infinite plane at an infinite distance. In other words, the camera 502 may be considered coincident with the shadow processing point 510, and any point along the reflection vector 512 may be directly projected into screen space to look up the camera feed texture. This approach is fast, simple, and may not require deferred shadow processing (unlike conventional implementations of screen space reflections), providing benefits for mobile devices where AR is prevalent.

[0067] In some embodiments, points projected outside the field of view of the camera 502 may initially be assigned texture coordinates outside the valid range (e.g., [0, 1]) and must be brought back into the valid range. There are several ways to bring them back into the valid range. In one exemplary embodiment, the two coordinates may be clamped independently to bring them into the valid range. In another exemplary embodiment, each coordinate may be mirrored when it exceeds the [0, 1] boundary and reflect the texture content horizontally, vertically, or on both. In another exemplary embodiment, the two coordinates may be adjusted so that they remain within the valid range region while retaining their orientation relative to the center point (0.5, 0.5). These techniques reasonably and inexpensively extrapolate colors in any direction from the center of the camera feed texture. Regardless of which technique is used to bring the out-of-bounds texture coordinates into the valid range, visible artifacts may be generated due to apparent pixel duplication. However, because the above mixing equation (2) decreases rapidly with increasing deflection of the reflection vector relative to the incident direction, such artifacts are usually not noticeable.

[0068] In some embodiments, in order to achieve proper blending between the environment map and the camera feed (that is, for proper blending of the environment map component 506 with the camera feed component 508), the brightness profiles of the environment map and the camera feed should roughly match. In other words, they should have visually similar brightness profiles. Because the camera feed can clamp the brightness values ​​of its pixels to a value of 1, and the brightness values ​​of the pixels of the environment map are significantly higher, in some embodiments, the camera feed can be scaled so that the brightest pixel of the camera feed matches a portion (e.g., half) of the brightest pixels of the environment map. Depending on the environment map, other adjustments / scaling ratios may also be effective, such as matching the mean, median, etc.

[0069] For example, in an environment map where the sun can be viewed directly, the camera feed can be scaled using percentiles around the mean or median. In another example, in an indoor environment map that covers a lower dynamic range of brightness, a portion of the maximum brightness can be used. In some embodiments, a final scaling factor based on local filtered camera brightness (and color) can be applied via false image based lighting (fIBL) so that the camera feed still controls the ultimate brightness of the mixed color to prevent, for example, reflections from becoming brighter than pixels that include the camera feed texture. In some embodiments, the filtered camera feed can be used to approximate the local lighting of an inserted computer-generated object. For example, a tone mapping operation on the camera feed can be used to satisfy (e.g., balance) two competing goals, which can include brightening the camera feed to match the cube. Figure 1 , and not brightening the camera feed to allow the specular reflections to appear significantly brighter than the camera feed pixels they appear to reflect.

[0070] In some embodiments, if the computing device 102 has multiple cameras (e.g., a binocular AR device), information from more than one camera (e.g., multiple cameras of the computing device 102) can be used to provide a wider field of view, and thereby allow valid camera feed component values ​​to be obtained in more directions, and thereby eliminate the need to fix out-of-bounds texture coordinates by mirroring. In an exemplary embodiment, this can be achieved by calculating homographies between all camera pairs. When a point is projected outside the field of view of the rendering camera, its homographies with other cameras can be applied to the projection. The homography that brings the projection into the field of view of the camera will allow the point to be sampled from the feed of that camera and blended into the final result for determining the camera feed component 508.

[0071] In some embodiments, available information about the depth of the surface depicted in the camera feed can be used to render more accurate screen space reflections, where the camera feed projection can be calculated at a specific depth along the reflection vector 506, rather than calculating the camera feed projection at infinite depth. This can be achieved in several ways. In one exemplary embodiment, if a small number of directional planes are provided, the reflection vector 506 can be intersected with each plane and the nearest intersection point accepted in the shader. In another exemplary embodiment, for more complex geometries, a depth map can be constructed and queried to find the nearest intersection point along the camera ray. Once the nearest intersection point is found, it can be projected into screen space and used for camera feed lookup as described above.

[0072] Figure 7An example of a computing device 700 and a mobile computing device 750 that can be used with the techniques described herein is shown. The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and a high-speed expansion port 710, and a low-speed interface 712 connected to a low-speed bus 714 and the storage device 706. Each of the above components 702, 704, 706, 708, 710, and 712 is interconnected using various buses and can be mounted on a general motherboard or otherwise installed as appropriate. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 704 or the storage device 706 to display graphical information for a GUI on an external input / output device (such as a display 716 coupled to the high-speed interface 708). In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, can be used as appropriate. Furthermore, multiple computing devices 700 may be connected, with each device providing a portion of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0073] The memory 704 stores information within the computing device 700. In one embodiment, the memory 704 is one or more volatile memory units. In another embodiment, the memory 704 is one or more non-volatile memory units. The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0074] The storage device 706 can provide mass storage for the computing device 700. In one embodiment, the storage device 706 can be or include a computer readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid state storage device, or a series of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more of the methods described above. The information carrier is a computer or machine readable medium, such as the memory 704, the storage device 706, or a memory on the processor 702.

[0075] The high-speed controller 708 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 712 manages less bandwidth-intensive operations. This allocation of functionality is merely an example implementation and is not meant to limit the implementations of the present disclosure described and / or claimed herein. In one implementation, the high-speed controller 708 is coupled to the memory 704, the display 716 (e.g., through a graphics processor or accelerator), and to the high-speed expansion port 710 that can accept various expansion cards (not shown). In an implementation, the low-speed controller 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or to a network device such as a switch or router, for example, through a network adapter.

[0076] As shown, computing device 700 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 720, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 724. In addition, it can also be implemented in a personal computer such as a laptop computer 722. Alternatively, components from computing device 700 can be combined with other components in a mobile device, such as computing device 750. Each such device can contain one or more computing devices 700, 750, and the entire system can be composed of multiple computing devices 700, 750 that communicate with each other.

[0077] Computing device 750 includes, among other components, a processor 752, a memory 764, an input / output device such as a display 754, a communication interface 766, and a transceiver 768. A storage device, such as a microdrive or other device, may also be provided to device 750 to provide additional storage. Each of the above components 750, 752, 764, 754, 766, and 768 are interconnected using various buses, and multiple components may be mounted on a common motherboard or otherwise appropriately mounted.

[0078] The processor 752 can execute instructions within the computing device 750, including instructions stored in the memory 764. The processor can be implemented as a chipset including separate chips with multiple analog and digital processors. For example, the processor can provide coordination of other components of the computing device 750, such as control of a user interface, applications running on the computing device 750, and wireless communications of the computing device 750.

[0079] The processor 752 can communicate with the user through the control interface 758 and the display interface 756 coupled to the display 754. For example, the display 754 can be a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other appropriate display technology. The display interface 756 may include appropriate circuits for driving the display 754 to present graphics and other information to the user. The control interface 758 can receive commands from the user and convert them to submit to the processor 752. In addition, an external interface 762 that communicates with the processor 752 can be provided to enable close-range communication between the computing device 750 and other devices. For example, the external interface 762 can provide wired communication in some embodiments, or wireless communication in other embodiments, and multiple interfaces can also be used.

[0080] The memory 764 stores information within the computing device 750. The memory 764 may be implemented as one or more computer-readable media, one or more volatile storage units, or one or more non-volatile storage units. An expansion memory 774 may also be provided and connected to the computing device 750 via an expansion interface 772, for example, the expansion interface 772 may include a SIMM (single in-line memory module) card interface. Such an expansion memory 774 may provide additional storage space for the computing device 750, or may also store applications or other information for the computing device 750. Specifically, the expansion memory 774 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Thus, for example, the expansion memory 774 may be provided as a security module for the computing device 750, and may be programmed with instructions that allow the computing device 750 to be used safely. In addition, security applications may be provided via a SIMM card and additional information, such as placing identification information on the SIMM card in a manner that cannot be stolen.

[0081] For example, the memory 764 may include flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more of the methods described above. The information carrier is a computer or machine readable medium such as the memory 764, the expansion memory 774, or a memory on the processor 752, which can be received, for example, via the transceiver 768 or the external interface 762.

[0082] The computing device 750 may communicate wirelessly via a communication interface 766, which may include digital signal processing circuitry when necessary. The communication interface 766 may provide communications in various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, LTE, 5G / NR, and the like. For example, such communications may be performed via a radio frequency transceiver 768. In addition, short-range communications may be performed, such as using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 770 may provide additional wireless data related to navigation and location to the computing device 750, which may be used appropriately by applications running on the device 750.

[0083] Computing device 750 may also communicate audibly using audio codec 760, which may receive voice information from a user and convert it into usable digital information. Audio codec 760 may also generate audible sounds for the user, such as through, for example, a speaker in computing device 750. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on computing device 750.

[0084] As shown, computing device 750 can be implemented in many different forms. For example, it can be implemented as a cellular phone 780. It can also be implemented as part of a smart phone 782, a personal digital assistant or other similar mobile device.

[0085] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be special purpose or general purpose, coupled to a storage system, at least one input device, and at least one output device to receive data and instructions and to send data and instructions to them.

[0086] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0087] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (LED (light emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor / screen) to display information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.

[0088] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0089] The computing system may include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other.

[0090] In some embodiments, Figure 7 The computing device depicted in FIG. 7 may include sensors that interface with an AR head mounted device / HMD device 790 to generate an augmented environment for viewing inserted content within a physical space. For example, including Figure 7One or more sensors on the computing device 750 or other computing devices depicted in the AR headset 790 can provide input to the AR headset 790, or generally, can provide input to the AR space. The sensors may include, but are not limited to, a touch screen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 750 can use the sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, which can then be used as input to the AR space. For example, the computing device 750 can be incorporated into the AR space as a virtual object such as a controller, a laser pointer, a keyboard, a weapon, etc. When the user incorporates the computing device / virtual object into the AR space, its position can allow the user to position the computing device so as to view the virtual object in the AR space in a certain way. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device like an actual laser pointer. The user can move the computing device left and right, up and down, in a circle, etc. and use the device in a similar manner to using a laser pointer.

[0091] In some embodiments, one or more input devices included in or connected to the computing device 750 can be used as input to the AR space. The input device may include, but is not limited to, a touch screen, a keyboard, one or more buttons, a touch pad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input functions, a game controller, or other connectable input devices. When the computing device is incorporated into the AR space, a user interacting with the input device included on the computing device 750 can cause specific actions to occur in the AR space.

[0092] In some implementations, the touch screen of the computing device 750 may be rendered as a touch pad in the AR space. The user may interact with the touch screen of the computing device 750. For example, in the AR head mounted device 790, the interaction is rendered as movement on the touch pad rendered in the AR space. The rendered movement may control a virtual object in the AR space.

[0093] In some embodiments, one or more output devices included on the computing device 750 can provide output and / or feedback to the user of the AR head-mounted device 790 in the AR space. The output and feedback can be visual, tactile, or audio. The output and / or feedback can include, but are not limited to, vibration, turning on, off, or flashing and / or flashing one or more lights or strobe lights, sounding an audible alarm, playing a ringtone, playing a song, and playing an audio file. The output device can include, but is not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a strobe light, and a speaker.

[0094] In some embodiments, the computing device 750 can appear as another object in a computer-generated 3D environment. User interactions with the computing device 750 (e.g., rotating, shaking, touching a touch screen, sliding a finger on a touch screen) can be interpreted as interactions with objects in the AR space. In the example of a laser pointer in the AR space, the computing device 750 appears as a virtual laser pointer in the computer-generated 3D environment. When the user manipulates the computing device 750, the user in the AR space sees the movement of the laser pointer. The user receives feedback from the interaction with the computing device 750 in the AR environment on the computing device 750 or the AR head-mounted device 790.

[0095] In some embodiments, the computing device 750 may include a touch screen. For example, a user may interact with the touch screen in a particular way that may mimic what is happening on the touch screen and what is happening in the AR space. For example, a user may use a pinch-type motion to zoom content displayed on the touch screen. This pinch-type motion on the touch screen may cause the information provided in the AR space to be zoomed. In another example, the computing device may be rendered as a virtual book in a computer-generated 3D environment. In the AR space, pages of the book may be displayed in the AR space, and a swipe of the user's finger on the touch screen may be interpreted as turning / flipping pages of the virtual book. As each page is turned / flipped, in addition to seeing the page content change, audio feedback may also be provided to the user, such as the sound of turning pages in the book.

[0096] In some embodiments, in addition to the computing device, one or more input devices (e.g., a mouse, a keyboard) may be rendered in the computer-generated 3D environment. The rendered input device (e.g., a rendered mouse, a rendered keyboard) may be used as a rendering device in the AR space to control objects in the AR space.

[0097] Computing device 700 is intended to represent various forms of digital computers and devices, including but not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely example embodiments and are not meant to limit the disclosed embodiments described and / or claimed herein.

[0098] A number of embodiments have been described. However, it will be appreciated that various modifications can be made without departing from the spirit and scope of the description.

[0099] In addition, the logic flow depicted in the accompanying drawings does not require the particular order shown, or the sequential order to achieve the desired result. In addition, other steps can be provided to the described process or steps can be removed therefrom, and other components can be added to the described system or removed therefrom. Therefore, other embodiments are within the scope of the appended claims.

[0100] Although certain features of the described embodiments have been described as described herein, many modifications, substitutions, changes and equivalents are now conceivable to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. It should be understood that they are given only by way of example and not limitation, and various changes in form and detail may be made. Except for mutually exclusive combinations, any part of the apparatus and / or method described herein may be combined in any combination. The embodiments described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different embodiments described.

Claims

1. A method for specular reflection, comprising: determining a first portion of the specular reflection associated with the computer-generated object based on a first contribution from an environment map component at a shading point of the computer-generated object; determining a second portion of the specular reflection associated with the computer-generated object based on a second contribution from an intersection of a reflection vector on a camera feed, wherein the camera feed corresponds to a plane facing a camera, and wherein the reflection vector originates at the shading point and extends through the camera feed; as well as The specular reflection associated with the computer-generated object at the shading point is determined based on a blend of the first portion and the second portion of the specular reflection.

2. The method of claim 1 , wherein the blending of the first and second portions of the specular reflection is based on an angle between a negative view vector associated with the camera feed and a surface normal vector perpendicular to a surface of the computer-generated object. The method of claim 1 , wherein the environment map component at the shadow processing point is determined based on an environment map of a surrounding environment. The method of claim 3 , wherein the environment map is based on a cube map.

5. The method according to claim 1, further comprising: determining whether light is scattered from a surface of the computer-generated object; as well as In response to determining that the light is scattered from the surface of the computer-generated object, an image of the camera feed is blurred prior to determining a component of the camera feed at the intersection. The method of claim 5 , wherein the blurring of the image is performed using a mipmap filter. 7 . The method of claim 1 , wherein the color of the pixel at the intersection point represents the color of a component of the camera feed at the intersection point mixed with the environment map component.

8. The method according to claim 1, further comprising: Independently clamps texture coordinates that are projected outside the valid range to within the valid range.

9. The method according to claim 1, further comprising: determining a first brightness representation of an environment map associated with the environment map component and a second brightness representation of the camera feed associated with a component of the camera feed at the intersection; as well as The second brightness representation of the camera feed is scaled to a ratio of the first brightness representation of the environment map so that the second brightness representation is visually similar to the first brightness representation.

10. The method according to claim 1, wherein: The blending is biased towards the component of the camera feed at the intersection point to achieve a higher angle between the negative view vector and the surface normal vector.

11. The method according to claim 1, wherein: The blending is biased towards the ambient feed component to achieve lower angles between negative view vectors and the surface normal vector.

12. A device for specular reflection, the device comprising: processor; as well as a memory comprising instructions configured to cause the processor to: determining a first portion of the specular reflection associated with the computer-generated object based on a first contribution from an environment map component at a shading point of the computer-generated object; determining a second portion of the specular reflection associated with the computer-generated object based on a second contribution from an intersection of a reflection vector on a camera feed, wherein the camera feed corresponds to a plane facing a camera, and wherein the reflection vector originates at the shading point and extends through the camera feed; as well as The specular reflection associated with the computer-generated object at the shading point is determined based on a blend of the first portion and the second portion of the specular reflection.

13. The apparatus of claim 12, wherein the blending of the first and second portions of the specular reflection is based on an angle between a negative view vector and a surface normal vector, the negative view vector being associated with the camera feed, and the surface normal vector being perpendicular to a surface of the computer-generated object.

14. The device according to claim 12, wherein: The environment map component at the shadow processing point is determined based on an environment map of a surrounding environment. The apparatus of claim 14 , wherein the environment map is based on a cube map.

16. The apparatus of claim 12, wherein the processor is further configured to: determining whether light is scattered from a surface of the computer-generated object; and In response to determining that the light is scattered from the surface of the computer-generated object, an image of the camera feed is blurred prior to determining a component of the camera feed at the intersection.

17. The apparatus of claim 12, wherein the color of the pixel at the intersection point represents the color of a component of the camera feed at the intersection point mixed with the environment map component.

18. The apparatus of claim 12, wherein the processor is further configured to: Independently clamps texture coordinates that are projected outside the valid range to within the valid range.

19. The apparatus of claim 12, wherein the processor is further configured to: determining a first brightness representation of an environment map associated with the environment map component and a second brightness representation of the camera feed associated with a component of the camera feed at the intersection; and The second brightness representation of the camera feed is scaled to a ratio of the first brightness representation of the environment map so that the second brightness representation is visually similar to the first brightness representation.

20. A non-transitory computer readable storage medium having computer executable program code stored thereon, the computer executable program code, when executed on a computer system, causing the computer system to perform a method comprising: determining a first portion of a specular reflection associated with a computer-generated object based on a first contribution from an environment map component at a shading point of the computer-generated object; determining a second portion of the specular reflection associated with the computer-generated object based on a second contribution from an intersection of a reflection vector and a camera feed, wherein the camera feed corresponds to a plane facing a camera, and wherein the reflection vector originates at the shading point and extends through the camera feed; as well as The specular reflection associated with the computer-generated object at the shading point is determined based on a blend of the first portion and the second portion of the specular reflection.

Images