Real-time synthesis in mixed reality

Abstract

The embodiment of the invention relates to real-time synthesis in mixed reality. A system may include a memory device to store instructions and data; and at least one processing device to execute instructions stored in the memory device to: receive a background image in the mixed reality view and a digital object to be synthesized onto the background image, generate a 2D boundary region for the digital object, select a version of the background image at a predefined resolution, and generate a 2D boundary region for the digital object. Overlying a 2D boundary region on the background image of the selected version and obtaining a set of color samples of pixels of the selected version along a perimeter of the 2D boundary region, and determining values for the one or more digital light sources based at least in part on the set of samples, one or more digital light sources are used to illuminate a digital object in a mixed reality view.

Description

[0001] Divisional Application Instructions

[0002] This application is a divisional application of the Chinese invention patent application with an international application date of May 31, 2019, which entered the Chinese national stage on December 7, 2020, with national application number 201980038854.X and named “Real-time Synthesis in Mixed Reality”. Technical Field

[0003] The present disclosure relates to image processing, and more particularly to real-time compositing of visual elements and background images in a mixed reality viewer. Background Art

[0004] When integrating 3D objects with real-world images in mixed reality, the objects can often be perceived as overlays and not part of the real world depicted in the image. As a result, 3D objects often appear unlit or static, which breaks the illusion that the object is part of the real-world environment. Summary of the invention

[0005] The present disclosure describes a system and method for dynamically compositing a digital object onto a background image. An exemplary system may include: a memory device for storing instructions and data; and at least one processing device for executing instructions stored in the memory device to receive a background image and a digital object to be composited onto the background image in a mixed reality view. The at least one processing device may also generate a 2D boundary region for the digital object, select a version of the background image at a predefined resolution, and overlay the 2D boundary region on the selected version, and obtain a set of samples of the color of the pixels of the selected version along the perimeter of the 2D boundary region. The at least one processing device may also determine, based at least in part on the set of samples, values ​​for one or more digital light sources used to illuminate the digital object in the mixed reality view.

[0006] An exemplary method may include receiving a background image in a mixed reality view and a digital object to be composited onto the background image, generating a 2D boundary region for the digital object, selecting a version of the background image at a predefined resolution, overlaying the 2D boundary region on the selected version of the background image, and obtaining a set of samples of the colors of pixels of the selected version along a perimeter of the 2D boundary region, and determining values ​​for one or more digital light sources used to illuminate the digital object in the mixed reality view based at least in part on the set of samples.

[0007] An exemplary computer-readable storage medium may include instructions that, when executed by one or more processing devices, cause the one or more processing devices to receive a background image and a digital object to be composited onto the background image in a mixed reality view. The instructions, when executed, may also cause the one or more processing devices to generate a 2D boundary region for the digital object, select a version of the background image at a predefined resolution, and overlay the 2D boundary region on the selected version, and obtain a set of samples of the colors of the pixels of the selected version along the perimeter of the 2D boundary region. The instructions, when executed, may also cause the one or more processing devices to determine, at least in part based on the set of samples, values ​​for one or more digital light sources used to illuminate the digital object in the mixed reality view. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0009] The present disclosure describes various embodiments that may be understood and fully appreciated in conjunction with the following drawings:

[0010] Figure 1 Example systems according to various embodiments are schematically illustrated.

[0011] Figure 2 Illustrated are example objects and example video frames into which the objects are to be composited, in accordance with various embodiments.

[0012] Figure 3 FIG. 1 illustrates a structure of a structure of a composite material according to various embodiments. Figure 2 Example objects and video frames.

[0013] Figure 4A further illustrates a perspective view of a camera of an example mixed reality viewer in accordance with various embodiments, and Figure 4B Illustrated is a composite view of an example 3D object and an example video frame captured by a camera.

[0014] Figure 5 Illustrated is an example light rig for an object to be composited in a mixed reality view in accordance with various embodiments.

[0015] Fig. 6A The diagram shows the top view or "bird's eye view" of Figure 5 Illustrated light rig and 3D space in which 3D objects are placed.

[0016] Figure 6B yes Fig. 6A Side view of the 3D space.

[0017] Fig. 7A illustrates a perspective view of local illumination of an example 3D object according to various embodiments, and Figure 7B The front view thereof is shown.

[0018] Figure 8 The diagram shows the Fig. 7A and Figure 7B Create a set of mipmaps from the sample background image.

[0019] Fig. 9A Illustrated is the selection of a texture refinement from a set of texture refinements to locally illuminate each 3D object in a mixed reality view in accordance with various embodiments.

[0020] Fig. 9B is a mixed reality view with video frames Fig. 9A Front view of two 3D objects.

[0021] Fig.10 An example sampling process for a revised 2D bounding box for an example object is illustrated in accordance with various embodiments.

[0022] Fig.11A Combining samples using different weightings to obtain a color value for each of two digital lights is illustrated according to various embodiments.

[0023] Fig. 11B The diagram shows Fig.11A The sample weighting used in computing the right light.

[0024] Fig. 11C The diagram shows Fig.11A The sample weighting used in computing the left light.

[0025] Fig. 12A and Fig. 12B Illustrated is the lighting of an example 3D object synthesized with a background image as a function of the light intensity in the background image in accordance with various embodiments.

[0026] Fig.13A and Fig. 13B Another example lighting of an example 3D object synthesized with a background image as a function of the light intensity in the background image is illustrated in accordance with various embodiments.

[0027] Fig.14A and Fig. 14B A general reflectance map and using an ambient color value to color the reflection and adjust the intensity are illustrated, respectively, according to various embodiments.

[0028] Fig.15A and Fig. 15B yes Fig.14A and Fig. 14B Magnified view of the tines of an example fork and a tire of an example vehicle.

[0029] Fig.16A and Fig. 16B Illustrated is a soft shadow effect according to various embodiments.

[0030] Fig.17 The composition of visual elements and background images according to various embodiments is schematically illustrated.

[0031] Fig.18 The diagram shows Fig.17 Schematic example of ellipse transformation of a 2D bounding box.

[0032] Fig.19 The diagram shows the Fig.18 The bounding ellipse generated in the example is sampled at various points.

[0033] Fig. 20 Illustration of combining different weights Fig.19 The illustrated samples are used to generate digital light values ​​for each of the example digital left light and the example digital right light.

[0034] Fig.21 Illustrated are tile refinement selections according to the size of example visual elements, according to various embodiments.

[0035] Fig. 22 Illustrated is the simultaneous synthesis of multiple visual elements of various sizes and positions within a viewpoint in accordance with various embodiments.

[0036] Fig.23 An overview of the operational flow of a process for calculating light color using sampled colors using a normalized weighted averaging technique according to various embodiments is illustrated.

[0037] Fig.24 An overview of the operational flow of a process for drawing 3D objects in a mixed reality view with shadows is illustrated in accordance with various embodiments.

[0038] Fig.25 A block diagram of a computer device suitable for practicing the present disclosure is illustrated in accordance with various embodiments.

[0039] Fig.26 An example computer-readable storage medium according to various embodiments is illustrated having a computer-readable storage medium configured to practice the methods described in FIGS. Fig.24 Instructions on various aspects of the procedures described in or relating to the . DETAILED DESCRIPTION

[0040] The present disclosure describes embodiments with reference to the drawings listed above. Those of ordinary skill in the art will appreciate that the description and drawings illustrate rather than limit the present disclosure, and that the drawings are generally not drawn to scale for clarity of presentation. Such skilled persons will also recognize that by applying the inventive principles contained herein, more embodiments are possible, and such embodiments fall within the scope of the present disclosure, which is limited only by the claims.

[0041] When a 3D object is integrated in a mixed reality view, the 3D object typically appears to be an overlay and is not part of the real world depicted in the background image. In order to better synthesize the digital 3D object onto the real-world image, in an embodiment, the device camera and sensors can be used to estimate the real-world scene lighting. Based on a set of samples of the background image acquired at points near the user-selected location of the digital 3D object, in an embodiment, the 3D lighting configuration of the object can be inferred in real time, which can approximate the real-world lighting conditions. Therefore, in an embodiment, the 3D object can be integrated with the image of the physical world in a convincing manner. Using algorithms according to various embodiments, even input from a monocular RBG sensor can be successfully used as a background image source. In other embodiments, the exemplary technology can also be scaled to multi-sensor devices.

[0042] In an embodiment, the digital 3D object may be depicted as shaded when placed in a darker area of ​​the background image, and as bright when placed in an area of ​​the background image with stronger lighting. Additionally, in an embodiment, color shading may be used to simulate indirect lighting, including light reflected from surrounding objects onto the 3D object in the background image.

[0043] In the following description, reference is made to the accompanying drawings forming a part thereof, wherein throughout the entire text, the same numerals (or the last two digits of the index numerals as appropriate) represent the same parts, and embodiments that can be practiced are shown therein by way of illustration. It should be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the following detailed description should not be considered restrictive, and the scope of the embodiments is limited by the appended claims and their equivalents.

[0044] The operation of various methods can be described in sequence as multiple discrete actions or operations in a manner that is most helpful for understanding the claimed subject matter. However, the order of description should not be interpreted as implying that these operations must be related to the order. In particular, these operations may not be performed in the order presented. The described operations can be performed in an order different from the described embodiments. In additional embodiments, various additional operations can be performed and / or the described operations can be omitted, split or combined.

[0045] For purposes of this disclosure, the phrase "A and / or B" means (A), (B), or (A and B). For purposes of this disclosure, the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0046] This description may use the phrases "in one embodiment" or "in an embodiment", which may refer to one or more of the same or different embodiments, respectively. In addition, the terms "including", "comprising", "having", etc. used in relation to the embodiments of the present disclosure are synonymous.

[0047] In addition, it should be noted that the embodiment can be described as a process depicted as a flow chart, a flow diagram, a data flow diagram, a structure diagram or a block diagram. Although the flow chart can describe the operation as a sequential process, many operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operation can be rearranged. When the operation of a process is completed, it can be terminated, but it can also have additional steps not included in (multiple) drawings. The process can correspond to a method, a function, a process, a subroutine, a subprogram, etc. When the process corresponds to a function, its termination can correspond to the function returning to the calling function and / or the main function. In addition, the process can be implemented by hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof. When implemented with software, firmware, middleware or microcode, the program code or code segment that performs the necessary tasks can be stored in a machine or computer readable medium. The code segment can represent any combination of a process, a function, a subprogram, a program, a routine, a subroutine, a module, a program code, a software package, a class or an instruction, a data structure, a program statement, etc.

[0048] As used hereinafter, including the claims, the term "circuitry" may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and / or memory (shared, dedicated, or group) that executes one or more software or firmware programs, a combinational logic circuit, and / or other suitable hardware that provides the described functionality, a part of, or includes any of the above. In some embodiments, the circuitry may be implemented by one or more software or firmware modules, or the functions associated with the circuitry may be implemented by one or more software or firmware modules.

[0049] As used hereinafter, including in the claims, the term "memory" may refer to one or more hardware devices for storing data, including random access memory (RAM), magnetic RAM, core memory, read-only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing data. The term "computer-readable medium" may include, but is not limited to, memories, portable or fixed storage devices, optical storage devices, wireless channels, and various other media capable of storing, containing, or carrying (multiple) instructions and / or data.

[0050] As used hereinafter, including in the claims, the term "computing platform" may be considered synonymous with computer device, computing device, client device or client, mobile phone, mobile unit, mobile terminal, mobile station, mobile user, mobile device, user equipment (UE), user terminal, machine type communication (MTC) device, machine to machine (M2M) device, M2M device (M2ME), Internet of Things (IoT) device, subscriber, user, receiver, etc., and may sometimes be referred to as the above in the following, and may describe physical hardware devices that are capable of sequentially and automatically performing a series of arithmetic or logical operations, which are equipped to record / store data on a machine-readable medium, and transmit and receive data from one or more other devices in a communication network. Furthermore, the term “computing platform” may include any type of electronic device, such as a cellular telephone or smart phone, a tablet personal computer, a wearable computing device, an automated sensor, a personal digital assistant (PDA), a laptop computer, a desktop personal computer, a workstation, a video game console, a digital media player, an in-vehicle infotainment (IVI) and / or in-vehicle entertainment (ICE) device, an in-vehicle computing system, a navigation system, an autonomous driving system, a vehicle-to-vehicle (V2V) communication system, a vehicle-to-everything (V2X) communication system, a handheld messaging device, a personal data assistant, an e-book reader, an augmented reality device, and / or any other similar electronic device.

[0051] Figure 1 A schematic diagram of one embodiment of a system 100 according to various embodiments is shown. The system 100 may include a mixed reality viewer 132, which may be stored, for example, in the mass storage device 118 of the computing device 122. The mixed reality viewer 132 may be loaded into the memory 128 and executed by the processor 130 of the computing device 122 to perform one or more of the methods and processes described in more detail below. In an embodiment, some tasks such as rendering of 3D objects, planes, etc. may be performed by a graphics processing unit (GPU) 131 under the control of the processor 130 or in conjunction with the processor 130.

[0052] In embodiments, mixed reality viewer 132 may generate mixed reality environment 134, which may be displayed to a user via a display device such as display device 136. As described in more detail below, virtual environment 134 may include one or more virtual objects, such as virtual object 1 140, virtual object 2 142, and background image 145. Such virtual objects may include one or more digital or virtual images, such as, for example, the following reference images: Figure 3 7 , 3D holographic objects, other virtual objects, and two-dimensional images may be selected and inserted into the mixed reality environment 134 by a user interacting with the mixed reality viewer 132 via the user interface 157. These digital objects may be selected from a set of digital objects stored in a digital object database 192, to which the computing device 122 may be directly coupled, as shown by link 121, or may be coupled to the network 116, for example, via link 123.

[0053] The mixed reality viewer 132 may also include a tracker 150 to track any virtual objects that may be inserted into the mixed reality environment 134 and determine their position, orientation, and scale, for example, as they may be moved by a user in the mixed reality environment 134. Hereinafter, the position, orientation, and scale of an object may sometimes be referred to as its "transform." In addition, the mixed reality viewer 132 may also include a compositor 155 that, given the position, orientation, and scale of the virtual objects determined by the tracker 150, may infer the local lighting of each virtual object present in the mixed reality environment 134, and determine the values ​​of one or more digital light sources or digital lights 147 to illuminate each such virtual object according to one or more of the methods and processes described in more detail below.

[0054] For example, the computing device 122 may include an optical sensor system 168, such as a camera. The optical sensor system 168 may acquire a real-time video feed, which may be the source of the background image 145, onto which one or both of the virtual object 1 140 and the virtual object 2 142 may be composited in the mixed reality environment 134. Alternatively, the computing device 122 may be connected to a remote optical sensor system 169 via the network 116, which may alternatively or additionally provide a real-time video feed to the mixed reality viewer 132, which may be used as the background image 145. Still alternatively, the background image 145 may be loaded into the mixed reality viewer 132 from the memory 128 or from a video / image database 193 connected to the computing device 122 via the network 116 via the link 123. In embodiments, the optical sensor systems 168 and 169 may include, for example, a monocular RBG sensor.

[0055] In embodiments, computing device 122 may take the form of a desktop computing device, a mobile computing device such as a smartphone, a laptop, notebook or tablet computer, a network computer, a home entertainment computer, an interactive television, a gaming system, or other suitable type of computing device. Fig.25 Additional details regarding components and computing aspects of computing device 122 are described in greater detail.

[0056] In embodiments, computing device 122 may be operatively connected to display device 136 using a wired connection, or may employ a wireless connection via WiFi, Bluetooth, or any other suitable wireless communication protocol. For example, computing device 122 may be communicatively coupled to network 116. Network 116 may take the form of a local area network (LAN), a wide area network (WAN), a wired network, a wireless network, a personal area network, or a combination thereof, and may include the Internet.

[0057] As described in more detail below, computing device 122 may communicate with one or more other computing devices, such as a server (not shown), via network 116. Figure 1 The illustrated example shows computing device 122 as a separate component from display device 136. It should be appreciated that computing device 122 may be integrated into display 136 in other examples.

[0058] As described above, when a 3D object is integrated in mixed reality, it may often appear to the viewer or user as an overlay rather than as part of the real world depicted in the background video frame or image. To address this issue, and therefore to better synthesize digital objects onto real-world backgrounds, in embodiments, device cameras and sensors may be used to estimate real-world scene lighting. Then, in embodiments, the 3D lighting configuration of the real-world background image may be inferred in real time and digitally replicated to approximate the real-world lighting conditions of the background image and applied to the synthesized 3D object.

[0059] In embodiments, digital objects such as, for example, 3D objects, etc. can thus be integrated with the physical world in a convincing manner. Algorithms according to various embodiments can operate quickly to support real-time applications, and can, for example, run using at least input from a monocular RBG sensor, while also being scalable to multi-sensor devices.

[0060] Thus, in an embodiment, a 3D object may be displayed as shaded in a mixed reality viewer when placed in a darker area, and displayed as bright when placed under stronger lighting. Additionally, in an embodiment, color shading may be used to simulate indirect lighting.

[0061] Figure 2 An example digital object 210, a 3D bird wearing shorts and headphones, and an example video frame 220 showing a room with a table and a plurality of chairs located next to a set of windows are illustrated. Note that, as described above, the video frame 220 may be, for example, a real-time image acquired by a camera, or, for example, a frame of a pre-existing video file. According to various embodiments, the example 3D bird object 210 may be composited onto a video frame showing a room. Note that, in Figure 2 In the example of , object 210 can be a 3D model that can be combined onto video frame 220. Alternatively, video frame 220 can be a frame in a previously stored video file or clip, which can, for example, be stored in a memory on a separate computing device, or can, for example, be received on the computing device from another computing device over a network. Still alternatively, video frame 220 can be a single image taken with a camera or image sensor, and is not necessarily part of a video clip at all. Here, it should be noted that, as used herein, the term "compositing" should be understood to refer to combining visual elements from separate sources into a single image to create the illusion of a single scene.

[0062] Figure 3 FIG. 1 illustrates a structure of a structure of a 3D image after being synthesized according to various embodiments. Figure 2 An example bird object 310 and an example video frame 320 are shown. Figure 3 In the example of , bird 310 is to be inserted into video frame 320 so that bird 310 appears to be standing in a room depicted in video frame 320, which in this example may be a frame of a live video feed. As described more fully below, to create a natural appearance, in embodiments, the lighting on 3D bird object 310 in the hybrid (synthesized) image may be adjusted to match the lighting in video frame 320.

[0063] Figure 4A Also illustrated are example camera perspectives for a mixed reality viewer in accordance with various embodiments. Figure 4BThe result of compositing in a mixed reality viewer is illustrated. In the case of a mixed reality viewer as shown in FIG. 4 , there may be two image sources, such as, for example, a live camera feed that generates video frames 425 acquired by camera 401 (not shown), as described above, and a virtual 3D object 410 inserted into the scene by a user of the MRV. Using tracking techniques such as, for example, a feature-based tracker, the 3D object 410, here a bird, may be inserted into the scene at a user-preferred location depicted by the live camera feed. The desired result is that the object seen by the user or viewer feels like it truly is part of the world captured through the eyes of camera 401. In an embodiment of the mixed reality viewer, the tracker may pay attention to the position, while a compositing module may be used to optimally visually integrate the object. The end result of the compositing is an image such as shown in image plane 435, as shown in the perspective view of camera 401, and as may be seen in FIG. Figure 4B The image shown in image 420 is viewed on the computing device screen.

[0064] It should be noted here that, in general, there are various techniques that can be used for synthesis. In fact, many techniques are used by filmmakers for the post-production of films. However, in the case of MRV, there may be several constraints, which, in an embodiment, tend to drive things that may or may not be completed, and therefore require different solutions. In an embodiment, these constraints may include, for example, real-time and limited computing resources (usually, in the example MRV application, trackers and cameras may have priority), viewing conditions, user movement (digital objects can go in and out of the frame), automatic adjustment of cameras or image sensors, RGB-only cameras (no depth sensor, no depth information outside the box). In an embodiment, real-time synthesis must operate under one or more of these constraints.

[0065] Typically, a user of MRV can define one or more digital light sources that are used to illuminate a digital object introduced into a given scene. A collection of these digital light sources may be referred to as a "light rig" associated with an object. As described in detail below, in an embodiment, each digital object to be inserted into an MRV frame may have its own light rig. Figure 5 An example light fixture is shown in accordance with various embodiments. Figure 5 In the example light rig of , there are two main directional light sources, right light 510 and left light 520, which are respectively positioned (virtually in the example MRV) on each side of the camera 501, at a lateral distance 503 from the camera 501 in the 3D space of the camera's perspective, and at a height 505 from the camera 501. As shown, each of the right light 510 and the left light 520 is directed toward the object 530 to be synthesized, here, Figure 2- The bird of Figure 4, shown from the front and from above. Figure 5 As further shown, 3D digital object 530 may be located on a virtual ground plane 550, shown here as a gray circle. In an embodiment, the virtual ground plane may be used to illustrate shader functionality, as described in detail below.

[0066] Continue to refer Figure 5 , in an embodiment, light sources 510 and 520 may preferably face the digital object in 3D space of the camera's perspective (e.g., in front of 3D digital object 530 as shown) so that the model never appears "flat," which would eliminate the sense of size. It should be noted here that in alternative embodiments, a fill light source providing light from below object 530 may be used, but be aware that this approach may sometimes bring in too much light from below the model. Therefore, such an approach is more unusual because, on average, most of the light comes from the top of a given object. As shown, the back limit of the camera's perspective is image plane 540, located behind 3D digital object 530. The net result of the 3D digital object placed in its 3D position and illuminated by right light 510 and left light 520 is composite image 535.

[0067] In addition to the light sources 510 and 520 described in more detail below, an ambient light source may also be provided, which may be used as an ambient tint. Thus, in an embodiment, the ambient light source does not have a specific position in the mixed reality view.

[0068] Fig. 6A Illustration from an overhead or "bird's eye" viewpoint Figure 5 The optical device of FIG6 has a width dimension of Figure 5 Thus, right light 610 and left light 620 are shown in the exemplary mixed reality view at their true virtual positions. Similarly, the position of camera 601 is shown as described above with respect to Figure 4A and Figure 5 The viewpoint is shown in a similar manner in front of the digital object 630, and also shows the image plane 635 at its position relative to the camera 610 and the two light sources 610 and 620. As shown, the 3D digital object 630 may be provided on top of a virtual ground plane 650.

[0069] Figure 6B yes Fig. 6A A side perspective view of the light installation shown in FIG. 1 , wherein reference numerals denote the same Fig. 6A The element described is the same as the element.

[0070] Described next 7A to 11C FIG. 1 illustrates a method for determining an example 3D object for compositing with a video frame in a mixed reality view according to various embodiments. Fig. 6A and Figure 6BExample process of the value of the light equipment shown.

[0071] refer to Fig. 7A , shows a perspective view of an example mixed reality viewer. Thus, Fig. 7A is a breakdown of what the user sees, which is Figure 7B The perspective view shows a 3D space 715, with an example video frame provided as a plane 710 behind the 3D space. In an embodiment, in the environment of the mixed reality viewer, a user can select a digital object and insert it into the 3D space 715. Thus, as Fig. 7A As shown, the example digital object 730 has been inserted into the mixed reality view as shown along with the video frame 710. As described above, the video frame 710 can be a frame in a real-time video feed acquired by a camera, such as, for example Figure 1 The optical sensor system 168 or 169. Alternatively, the video frame may be a previously acquired image loaded from memory into the example mixed reality viewer. Fig. 7A , an example digital object 730 may be a white bird with headphones and light blue shorts, and as Fig. 7A As shown, a user may be placed in a 3D space 715 of an exemplary mixed reality viewer, which may be, for example, Figure 1 A mixed reality viewer 132 is provided.

[0072] As described above, in embodiments, a tracker that is also part of the mixed reality viewer may be used to calculate the relative orientation and position of the digital object 730 within the 3D space 715. Once the user, such as via Figure 1 This may occur if the user interface 157 of the user interface 157 of the user interface 157 is used to place the digital object there. Once the position and orientation of the digital object 730 are obtained from such a tracker, in an embodiment, a 3D bounding box 735 can be calculated around the dimensions of the digital object 730. Fig. 7A , 3D bounding box 735 is shown in yellow. As shown, 3D bounding box 735 tracks the orientation of 3D object 730, which is rotated about 30 degrees to the left of the figure (from a front view perspective) relative to video frame 710, so that the bird is turned to its right.

[0073] In an embodiment, the 2D bounding area 720 may be calculated from the outer dimensions of the 3D bounding box 735. Fig. 7AAs shown, the 2D boundary area 720 can be a projection of the 3D bounding box 735 on the provided 2D plane 718, which is in front of the 3D space 715 and parallel to the video frame 710. The 2D plane 718 may be referred to as a "projection plane" or "image plane" hereinafter. It should be noted that although such a projection of the 3D bounding box 735 on the 2D plane 718 may cause a slightly larger 2D boundary area than a tight fit, as described below, the inventors believe that it works well in terms of color sampling. Note that the reason such a projection may cause a slightly larger 2D boundary area is that the 3D bounding box itself is a conservative approximation of the boundary of the true shape of the example object. Therefore, the volume of the 3D bounding box is always larger than the volume of the actual 3D object. Therefore, the projection of its volume on the image plane generally causes a larger area than the "perfect" projected image of the object. Note that the 2D boundary area 720 allows the collection of color information close to the 3D object. For example, by projecting the points on the 2D boundary area 720 back into the 3D space, the resulting points in the 3D world are more likely to be in the 3D neighborhood of the object.

[0074] Figure 7B Presented Fig. 7A A front view of a mixed reality image, as would normally be seen by a user of a mixed reality viewer. Fig. 7A , the 2D bounding region 720 is shown as the outer boundary of the surrounding 3D bounding box 735 as projected onto the 2D screen space 718.

[0075] Figure 8 Illustrated is the creation of a texture refinement from a background image or video frame according to various embodiments. Figure 8 , a set of map refinements 810 is shown. Note that a set of map refinements refers to a sequence of optimized images, each of which is a progressively lower resolution representation of the same image. For example, in Figure 8 In a texture refinement of , each texture refinement to the left of its "parent" averages the colors of a 2x2 pixel volume of the parent texture refinement into one color value. In embodiments, by using texture refinements, noise can be removed because lower resolution texture refinements are less prone to changing rapidly. In embodiments, this may be the case both spatially and frame-to-frame. With respect to space, as the model moves across the screen, the resulting lighting may be smoother because the texture refinement tends to act as a smoothing function over areas of pixels. Similarly, with respect to frame-to-frame changes, if the background itself changes, sudden changes in some pixels can, for example, be averaged with the rest of the pixels, thereby avoiding sudden changes in lighting. Of course, it should be understood that if a large portion of the background changes suddenly, such as, for example, if the light is turned off in a real-world scene captured in real time, then the texture refinement will have a harder time smoothing the changes as expected.

[0076] Continue to refer Figure 8 , the highest resolution version of video frame 710 (from FIG. 7 ) is Figure 8 The rightmost and lowest resolution are Figure 8 In an embodiment, a texture refinement may be selected for each 3D object, and its color value may be sampled at a set of points surrounding the 3D object. This process is described next.

[0077] Fig. 9A and Fig. 9B Depicts texture refinement options according to various embodiments. Fig. 9A and Fig. 9B , note that the 2D area 920 calculated as described above in conjunction with FIG. 7 can be used to determine which map refinement to use for color sampling to determine the light rig for the 3D object in the mixed reality synthesis. By sampling the colors of the map refinement along the periphery of the boundary area of ​​the 3D object, the local lighting near the 3D object can be inferred. By doing so, since no undersampling is performed, the pixel-averaged nature of the map refinement can be exploited, which prevents loss of information. In addition, in embodiments, this use of map refinement can provide more predictable lighting when scaling the 3D model, and can be used when the viewpoint of the 3D object changes, such as when, for example, Fig. 7A As the user is shown moving (via translation and / or rotation) the 3D object through the 3D space 715, this allows the lighting to update correctly, or even scale the object as it may move to achieve a close-up or far-away effect.

[0078] like Fig. 9A and Fig. 9B As shown, in this example, there are actually two 3D objects inserted into the mixed reality view of the video frame 910. The two 3D objects are bounded by 2D boundary regions 920 (the larger one is the foreground) and 925 (the smaller one is the background) as shown. Because the 2D boundary region 920 is significantly larger than the 2D boundary region 925, to sample the colors near the 2D boundary region 920, a lower resolution map refinement can be selected, such as the map refinement 950 shown in the figure. On the other hand, given a smaller size of the 2D boundary region 925, in order to have enough color samples along the periphery of the 2D boundary region 925, a significantly higher resolution map refinement can be selected, such as the map refinement 955.

[0079] Fig.10An example sampling of map refinements in an area of ​​a 3D object is illustrated in accordance with various embodiments. As described above, in an embodiment, the size of the 2D boundary area in the projection plane can be used to select an appropriate map refinement for sampling. In an embodiment, in order to provide sufficient granularity, the map refinement should have multiple samples (pixels) along each dimension of the 2D boundary area. In an embodiment, as described above, this is a function of the size (at a selected scale) of the 3D object within the 3D space of the mixed reality viewer for a given video frame. Fig.10 In the example of , there are 4 pixels along the horizontal dimension of the 2D border region 1020, and 5 pixels along the longer vertical dimension. In an embodiment, for example, the shorter dimension of the 2D border region may have at least 4 pixels.

[0080] In an embodiment, a fitting boundary region closer to the 3D object can be generated from the rectangular 2D boundary region to perform actual color sampling. Fig.10 As shown, in an embodiment, an ellipse 1021 can be generated from a rectangular 2D boundary area 1020 to sample the color of the pixel of the selected map refinement that is actually closer to the object. In an embodiment, bilinear filtering can then be used to sample the texture at an integer number of positions around the boundary area (here, ellipse 1021) that is more closely fitted. In an embodiment, N can be equal to 8 as shown in the figure, thereby obtaining samples 1031, 1032, 1033, 1034, 1035, 1036, 1037 and 1038. Alternatively, N can be equal to 12, 16 or other suitable values. It should be noted here that in this context, the map refinement selection is associated with the number of samples that can be obtained as described above. Therefore, in an embodiment, the following formula can be used to calculate the sampling range and select appropriate map refinement. Initially, in an embodiment, the sampling range can be calculated by the following:

[0081] Sampling range = (sample count / 2) + 1,

[0082] It is assumed that the samples are evenly distributed around the perimeter of the boundary region, such as an ellipse. Then, the appropriate background map refinement can be selected using the following equation:

[0083] Sampling map level = max(log2(screen border.width / sampling range),

[0084] log2(screen border.height / sampling range)),

[0085] It is assumed that the screen bounds (ScreenBounds) are in the "background pixels" (Bg0).

[0086] It should be noted here that "color" in texture sampling is an umbrella term that includes chroma and intensity. Typically, textures can have multiple formats. For example, if a camera with high dynamic range (HDR) capabilities is used to acquire a background image and an HDR display, the values ​​of the color samples may exceed the conventional 8-bit color range per channel. In such an example embodiment, the RGB triplet can represent intensity and chroma (as a low dynamic range RGB triplet) by premultiplying the intensity with each channel. Alternatively, if the values ​​are to be separated, weighting can be applied as follows: weighted sample color = (sample.RGB * sample.intensity) * sample weight.

[0087] Thus, in an embodiment, a set of color samples may be obtained along the periphery of the ellipse 1021. From these samples, one or more digital light sources may be obtained, such as, for example, the left light and the right light described above with reference to the light rig of Figure 6. Once these values ​​are obtained, the rendering pipeline of the example mixed reality viewer may combine the color of the object with the color of the light from each digital light source.

[0088] Described next FIG. 11A to FIG. 11C The diagram illustrates combining samples using different weightings to obtain a color value for each of two digital lights according to various embodiments. Fig.11A In an embodiment, the left light 1151 and the right light 1150 can each use the sum of weighted samples to combine eight color samples 1131-1138 obtained along the periphery of the boundary ellipse 1121, where it is assumed that the sum of the weights of each light is equal to 1. For example, the digital light can generally be calculated using the following equation:

[0089]

[0090] Continue to refer Fig.11A Note that the six samples at the bottom of ellipse 1121, namely samples 1131, 1132, and 1133 at the bottom right and 1137, 1138, and 1131 at the bottom left, are weighted with zero weighting depending on the light being calculated (e.g., right or left). Fig. 11B , samples 1131, 1138, and 1137 are weighted with zero. The remaining five samples have positive weights, with the largest weight being given to sample 1134, which in this example is closest to right light 1150. Note here that in other embodiments, more than two digital lights may be calculated. Furthermore, regardless of how many lights are calculated in total, the position and orientation of each light may be defined by the program or a user of the program within a 3D space such as 3D space 715 of reference FIG. 7 .

[0091] Thus, according to various embodiments, which sample along a given 2D boundary shape will have a greater or lesser weighting is generally a function of the orientation and position of that sample within the 3D space of the digital light for which it is input. Fig. 11C , shows the weights of the samples that will be used to calculate the color of the left light 1151. Therefore, Fig. 11C Picture shows Fig. 11B The opposite case, here again, the samples farthest from the left light 1151, namely samples 1131, 1132 and 1133, have a weight of 0, and the sample closest to the left light 1151, sample 1136, has the largest weight, namely 0.555. It should be noted that the samples 1131, 1132 and 1133, respectively, are weighted to 0.555. Fig. 11B and Fig. 11C From the calculated colors of each of the right light 1150 and the left light 1151 in FIG. 1 , it can be seen that the right light 1150 has a larger red color input from the main sample 1134 , and the left light 1151 has a larger brown chromaticity due to the main sample 1136 .

[0092] Fig. 12A and Fig. 12B The lighting of an example 3D object synthesized with a background image is illustrated as a function of the dominant light in the background image at the time, according to various embodiments. Fig. 12A , an example 3D object that may be referred to as a monster is provided in Fig. 12A and Fig. 12B On the right side of each of them, each of those composite images is illuminated differently, depending on the prevailing lighting conditions of the background image at the time. It should be noted that Fig. 12A and Fig. 12B The two background images shown have the same location in the shopping mall, but were taken under two different actual lighting conditions. Fig. 12A brighter, and therefore the monsters are brighter, Fig. 12B Darker, and the monsters are correspondingly darker.

[0093] Fig.13A and Fig. 13B Another example lighting of an example 3D object synthesized with a background image is illustrated as a function of the light intensity in the background image in accordance with various embodiments. Fig.13A and Fig. 13B In the case of 3D objects, it is not the actual lighting that changes between the two images, but rather the position where the 3D object is placed relative to the background image that changes. Fig.13A , the example "red blob monster" 3D object 1320 is illuminated and colored differently depending on whether it is placed above a blue trash can, as shown in FIG. Fig.13A As shown, the blue trash can or its interior is a local darker part of the background image, such as Fig. 13B shown.

[0094] Fig.14A and Fig. 14B The overall reflection map and the use of the ambient color value to color the reflection and adjust the intensity are respectively illustrated according to various embodiments. Fig.14A The use of an overall reflectance map as implemented in usually refers to a single reflectance map, which works well in neutral and well-lit environments. However, note that real-world environments are not so predictable. Therefore, Fig. 14B The use of ambient color values ​​to colorize reflections and adjust intensity according to various embodiments is illustrated. In embodiments, as described above, the ambient color may be the average of all color samples obtained along the periphery of the example boundary region, such as, in embodiments, as described above in conjunction with Figures 10 to 11C The described inscribed ellipse derived from the 2D rectangular bounding box can be used to color the reflection and adjust the intensity. In an embodiment, this can help dim white highlights in dark scenes and help the reflection color match the environment.

[0095] Continue to refer Fig.14A , shows an example red car 3D object 1440 placed near a metal fork real-world object 1430, both of which are on a blue table. Fig. 14B The same two objects are shown, but placed on a set of pink cushions, such as those provided on outdoor furniture or the like. Fig.15A and Fig. 15B They are Fig.14A and Fig. 14B An enlarged view of the tire rim 1541 and the tip of the fork 1530 of the red car 1540. Fig.15A and Fig. 15B , by comparison, although Fig.15A The fork 1530 in the figure or the tire rim 1541 of the red car 1540 is not obviously colored, but in Fig. 15B In , each of these objects has a pink tint to simulate the reflection of ambient light at its respective location. In particular, in this example, the ambient light color is applied to the reflection contribution by tinting the reflection map used to illuminate the example 3D object. In an embodiment, as described above, the ambient light value can be generated by averaging all samples in their respective boundary regions (i.e., no relative weighting is applied).

[0096] In an embodiment, in addition to dynamic lighting of 3D digital objects in mixed reality, shadow effects may be generated for each 3D object to further enhance the synthesis. Fig.16A and Fig. 16B Illustrated is an example soft shadow effect for each of two 3D objects added to a real-world background image, here a desktop with a power strip adjacent to a computer display base, in accordance with various embodiments.

[0097] refer to Fig.16A and Fig. 16B , in an embodiment, Fig.16A Motorcycles and Fig. 16B For each of the superheroes, the two 3D digital objects can have a shadow digitally added to the composite image as shown, so that the 3D digital objects appear to have local illumination of the background image at their respective locations. By comparison, the real-world wires seen directly behind each 3D object cast actual shadows on the desktop, and the light appears to emanate from the desktop. In this example, the actual real-world light comes from the office lighting above, which is an LED lamp. Therefore, in an embodiment, a custom shader (shadow simulator) can be implemented that can simulate directional light from the top and can assume a primarily diffuse light environment. In an embodiment, this can be implemented by an example mixed reality viewer according to various embodiments, which passes the vertical light direction to the shader parameters. In addition, when the distance from the shadow projector to the ground plane is very large, the diffuse aspect can be represented by adjusting the shader to blur the shadow. Note here that in an embodiment, these parameters can be controlled by a given application and can therefore be changed dynamically or from an algorithm-based decision process.

[0098] In addition, in embodiments, custom shaders may utilize blur and intensity with distance attenuation, and in embodiments, may not be as dynamic as the lighting of a 3D object using an example light rig as described above. Thus, in such embodiments, the custom shader may not change shadow casting based on the direction of real-world lighting or the real-world shadow color. Note, however, that in other embodiments, these shadow parameters may be changed dynamically, and the example shader process provided below does not preclude such dynamic changes to shader parameters. In embodiments, a single neutral color may be used in a custom shader, such as, for example, gray with equal R, G, and B values, and thus their values ​​only drive the intensity of the color.

[0099] In an embodiment, a custom shader may implement the process described in the following pseudo code:

[0100] Given the following input:

[0101] 3D Model

[0102] Virtual Ground Plane

[0103] 3D rendering system

[0104] algorithm

[0105] A. Calculate the 3D boundary of the 3D model (model boundary);

[0106] B. Set the virtual camera (C阴影 ) is placed outside the 3D model boundary (model boundary) and below the virtual ground pointing upwards, the virtual camera is set to select only the more

[0107] Near depth value;

[0108] *(The direction does not have to be completely vertical, any upward direction will do)

[0109] (Alternative version: Place a virtual camera above the model pointing downward, but select only its

[0110] his depth value);

[0111] C. Draw the depth component of the 3D object from the viewpoint and store it in addressable storage

[0112] M 阴影 (e.g. texture memory);

[0113] D. Set another virtual camera (C 查看器 ) is placed at the position from which the viewer views the 3D object;

[0114] E. Draw 3D objects normally; and

[0115] F. Draw the ground plane using the following algorithm:

[0116] For each point on the ground plane (P 地 ):

[0117] a. From the viewpoint (D 地 )Calculate (P 地 ) and (C 阴影 )

[0118] b. Calculate M 阴影 (P 地 ) coordinates and obtain the corresponding 3D model depth value (D 模型 )

[0119] c. Calculate D delta =D 地 -D 模型 The difference between

[0120] d. Use model boundary normalization D delta

[0121] e. Use D delta Amplitude is calculated:

[0122] i. Shadow intensity, and

[0123] ii. Shadow blur radius.

[0124] It should be noted here that the closer the resolution match between the background image and the visual elements, the more realistic the composite image appears. Therefore, a high-resolution object composited onto a low-resolution camera frame may look out of place, and sometimes look severely out of place. In an embodiment, by adjusting the pixel resolution of the drawn object to match the resolution of the camera, the drawn object can appear more immersed in the real world as seen through the camera, and thus the mixed reality view is more realistic.

[0125] Therefore, in an embodiment, the resolution matching algorithm may be implemented according to the following pseudo code:

[0126] Given the following input:

[0127] Background 2D Image

[0128] Wb: background width in pixels

[0129] Hb: background height in pixels

[0130] 3D Model

[0131] 3D rendering system with 2D output

[0132] Wr: draw output width in pixels

[0133] Hr: Draw output height in pixels

[0134] 2D display system (such as a screen or any software window)

[0135] Wd: Displays the horizontal resolution in pixels

[0136] Hd: Displays the horizontal resolution in pixels

[0137] algorithm:

[0138] 1. Set the rendering system output resolution to match the background resolution;

[0139] Wr=Wb;

[0140] Hr=Hb.

[0141] 2. Draw 3D models and backgrounds using any known compositing techniques;

[0142] 3. Output the rendered image to a 2D display using any existing image size remapping technique (e.g., cropping or stretching) with or without preserving the aspect ratio of the rendered output.

[0143] With respect to the example resolution matching pseudo-code provided above, it should be noted that in embodiments, it is perfectly acceptable for the example rendering system to use anti-aliasing features internally to enhance the quality of the output, as long as the final output is resolved to match the constraints Wr=Wb and Hr=Hb.

[0144] Described next Figures 17 to 22 Provided as above with reference to Figures 7 to Fig. 11C A sequential diagram of the illustrated and described dynamic lighting of a 3D object in a mixed reality viewer. The sequence of diagrams is followed by example pseudo code for the illustrated process.

[0145] refer to Fig.17 , the mixed reality view may include a background image 1701 and one or more visual elements 1705. The visual element 1705 may be 3D as described above, for example, or may be 2D in nature, being a bounded plane at a certain location and orientation within the 3D space of the mixed reality environment. The visual element 1705 may be bounded by a 2D bounding box 1710, which may be used as a local sample area to obtain samples of the background image to infer the lighting at the location of the visual element.

[0146] Fig.18 illustrates an ellipse transformation of a 2D bounding box 1810 of a visual element 1805 to an ellipse 1811, and Fig.19 The diagram shows the Fig.18 The bounding ellipse 1911 generated in the illustrated ellipse transform samples the color of the background image 1901 at various sampling points 1920 , for example eight (8) sampling points, which closely surround the visual element 1905 .

[0147] Fig. 20 The diagram shows the use of different weights (not shown) to combine Fig.19 The illustrated samples are used to generate digital light values ​​for each of the example digital left light 2015 and the example digital right light 2025. In general, all of the samples 2031 to 2042 can be combined with appropriate weighting to obtain a final color value for both lights in the light rig. However, in some cases, some samples may have their illumination set to zero due to being relatively far away from a given light, and therefore they will not contribute to the final color value of the light. Fig. 20, where for each light, the unused samples are depicted as being outside of the dashed ellipse for each light. Thus, the left light 2015 does not use the values ​​of samples 2032, 2033, or 2034 that are shown as not within the dashed ellipse 2010, and similarly, the right light 2025 does not use the values ​​of samples 2040, 2041, or 2042 that are shown as not within the dashed ellipse 2020. Note that in embodiments, the number of samples whose weights are set to zero need not be the same for each light source, although this is the case here because the right light 2015 and the left light 2025 are symmetrically positioned relative to the visual element 2005.

[0148] Fig.21 The selection of texture refinements according to the size of an example visual element according to various embodiments is illustrated. Thus, for a small visual element 2105, a higher resolution texture refinement 2110 of the background image is required so that the visual element covers a sufficient number of texture refinement pixels to have a sufficient number of samples in the local area of ​​the texture refinement to capture the variation in light color as described above. Similarly, for a large visual element 2106, a lower resolution texture refinement 2150 of the background image may be used because, given the size of the large visual element, even at a lower resolution, there is a sufficient number of texture refinement pixels and therefore a larger pixel size in the texture refinement is covered by the visual element.

[0149] Fig. 22 The illustration shows the simultaneous synthesis of multiple visual elements, such as, for example, small visual elements 2208 and 2209, larger visual elements 2206 and 2207, located at the front of the 3D space of the mixed reality viewer, and a medium-sized visual element 2205 (bear) located at the back of the 3D space, which is located on top of an object depicted in the distance of the background image 2201. According to various embodiments, all visual elements can be synthesized onto the background image 2201 on a per-visual element basis using the dynamic lighting techniques described above. It is noted here that in embodiments, there is no limit to how many visual elements can be supported. However, an upper bound may be what a given computing device can calculate within the allocated time budget that the example mixed reality viewer application is given to calculate a frame.

[0150] The following pseudo code represents the Figures 17 to 22 The process schematically presented in and the process described above are more generally referred to as:

[0151] Given the following input

[0152] Background (image) (Bg)

[0153] Visual elements (Ve)

[0154] Visual element transformation, PRS = {position, rotation, scale}

[0155] Sample Count(Sample Count)

[0156] Light count, the number of lights used to illuminate a visual element

[0157] A.

[0158] Generate background (Bg) map refinement [1]

[0159] Texture refinement (Bg) = {Bg0, Bg1, Bg2, ... BgN}

[0160] B.

[0161] For each visual element (Ve)

[0162] a. Calculate the Ve boundary 3D shape in Ve space (VeBounds)

[0163] (bounding box, bounding sphere, convex volume, bounding box hierarchy...);

[0164] b. Calculate the background 2D screen space bounds (screen bounds) by projecting VeBounds onto the screen

[0165] (If necessary, the screen boundaries need to be constrained within the screen boundaries);

[0166] c. Calculate the ellipse (E) drawn within the screen boundaries;

[0167] d. Calculate the sampling range using the following formula:

[0168] Sampling range = (sample count / 2) + 1

[0169] *(Assuming samples are evenly distributed around the perimeter of the ellipse)

[0170] e. Use the following formula to select one of the background map refinements

[0171] Texture refinement level = max(log2(screen border.width / sampling range),

[0172] log2(screen border.height / sampling range))

[0173] *(Assuming the screen borders are in "background pixels" (Bg0))

[0174] f. Sample the selected background map using elliptical distribution samples

[0175] For each (i in [1, sample count])

[0176] {

[0177] Sample Color i = Sample function (Bg 贴图细化级别 , sample location i )

[0178] }

[0179] *(The sampling function is a bilinear image sampling function)

[0180] g. Calculate the light color using the sampled colors using a normalized weighted average technique:

[0181] for each (i in [1, light count])

[0182] {

[0183]

[0184] }

[0185] *(Weights are application specific and are usually defined based on sample and light properties)

[0186] *(Assume that the sum of each light's weights equals 1 (normalized weighted average))

[0187] Reference now Fig.23 , presents an overview of the operational flow of a process 2300 for calculating light color using sampled colors using a normalized weighted average technique according to various embodiments. The process 2300 is similar to that described above. Figures 17 to 22 Note that process 2300 is an example of a general process expressed by the pseudo code provided above according to various embodiments. According to various embodiments, process 2300 may be performed by, for example, Figure 1 , wherein the example mixed reality viewer 132 may be run therein. Process 2300 may include blocks 2310 to 2370. In alternative embodiments, process 2300 may have more or fewer operations, and some operations may be performed in a different order.

[0188] Process 2300 may begin at block 2310, where an example device may receive a background image in a mixed reality view and a transformation of a digital object relative to the background image. As described above, the transformation may include a transformation performed by a user in an example mixed reality viewer (such as, for example, Figure 1 The values ​​of the position, rotation (orientation), and scale of a digital object positioned in a 3D space generated by a mixed reality viewer 132 of the 3D image processing system. In an embodiment, the transformation of the digital object can be generated by a tracker (such as, for example, Figure 1 determined by the tracker 150).

[0189] From box 2310, process 2300 may proceed to box 2320, where a set of pre-filtered versions of the background image at different resolutions may be generated. As described above, in an embodiment, this may include a set of map refinements. From box 2320, process 2300 may proceed to box 2330, where a 2D bounding box for the digital object may be calculated in the screen space of the mixed reality viewer. This may be a two-step process, for example, as described above. Initially, a 3D bounding box for the digital object may be calculated within the 3D space generated by the mixed reality viewer. Then, a projection of a 2D bounding box containing all edges of the 3D bounding box may be generated in the screen space of the mixed reality viewer from the 3D bounding box.

[0190] From block 2330, process 2300 may proceed to block 2340, where the 2D bounding box may be transformed to a more tightly fitting bounding region. For example, as described above, a rectangular 2D bounding box may be transformed to an inscribed ellipse. In alternative embodiments, other shapes or types may be used where the shape of the digital object may better fit another bounding region. In embodiments, such a transformed and more tightly fitting bounding region may be used for sampling of a background image.

[0191] Process 2300 may proceed from block 2340 to block 2350, where one of a set of pre-filtered versions of the background image may be selected for sampling. Fig. 22 As described, for example, the resolution of the background image may be selected based on the size of the digital object, in order to ensure that a sufficient number of pixels along each dimension are covered by the digital object, while selecting the lowest possible resolution.

[0192] Process 2300 can proceed from box 2350 to box 2360, where a set of color samples of the selected version of the background image can be obtained along the perimeter of the more tightly fitting boundary area. For example, as described above, eight samples can be obtained at points that are equally distanced along the ellipse. Finally, process 2300 can proceed from box 2360 to box 2370, where values ​​of one or more digital light sources in the mixed reality view can be determined based at least in part on the set of samples. For example, each of the one or more digital light sources can be a weighted combination of a set of color samples. In some embodiments, the weight of each sample can be a function of its spatial proximity to the digital light source. For example, as described above with reference to Fig. 11B and Fig. 11C As described, the values ​​of the right and left lights may be determined from the set of eight samples, and the weighting of the eight samples for each light may be a function of the distance of a given sample from the particular light.

[0193] Reference now Fig.24, presents an overview of the operational flow of a process 2400 for providing shadows for a 3D digital object according to various embodiments. The process 2400 is similar to the above reference Fig.16A and Fig. 16B According to various embodiments, process 2400 may be performed by, for example, Figure 1 2400. However, it should be noted here that a custom shader that may implement process 2400 may not need to sample or receive as input a background image on which a 3D object is to be synthesized as described above. Instead, process 2400 only requires a virtual ground plane as input. In an embodiment, for example, a virtual object may be used to draw a shadow in a planar shape. For example, a virtual plane may be drawn using a pixel shader that may, for example, calculate each pixel value of the plane as it would be seen from a user's viewpoint. Thus, in an embodiment, the same computing device that draws the 3D object may be used to draw the shadow, such as, for example, Figure 1 GPU 131 is shown.

[0194] Continue to refer Fig.24 , process 2400 may include blocks 2410 to 2470. In alternative embodiments, process 2400 may have more or fewer operations, and some operations may be performed in a different order. Process 2400 may begin at block 2410, where an example apparatus may receive or access a 3D object, a virtual ground plane, and a 3D rendering system. Process 2400 may proceed from block 2410 to block 2420, where a boundary of the 3D object may be calculated. Process 2400 may proceed from block 2420 to block 2330, where a virtual camera may first be placed outside the boundary of the 3D object, below the virtual plane, and pointing upward.

[0195] Process 2400 may proceed from block 2430 to block 2440, where the depth component of the 3D object may be rendered from the viewpoint of the virtual camera and stored in a memory such as, for example, Figure 1 The process 2400 may proceed from block 2440 to block 2450, where a second virtual camera may be placed at the user's viewpoint, and the process 2400 may proceed from block 2450 to block 2460, where a 3D object may be drawn from the viewpoint of the second virtual camera.

[0196] Finally, process 2400 may proceed from block 2410 to block 2320, where the above reference may be used, for example. Fig.16A and Fig. 16B The provided custom shader algorithm draws the ground plane point by point.

[0197] Reference now Fig.25, wherein a block diagram of a computer device suitable for practicing the present disclosure according to various embodiments is illustrated. As shown, the computer device 2500 may include one or more processors 2502 and a system memory 2504. Each processor 2502 may include one or more processor cores and a hardware accelerator 2575. Examples of hardware accelerators 2505 may include, but are not limited to, programmed field programmable gate arrays (FPGAs). In an embodiment, the processor 2502 may also include a memory controller (not shown). In an embodiment, the system memory 2504 may include any known volatile or non-volatile memory.

[0198] In addition, the computer device 2500 may include (multiple) mass storage devices 2506 (such as solid-state drives), input / output device interfaces 2508 (to interface with various input / output devices such as a mouse, cursor control, display devices (including touch-sensitive screens), and communication interfaces 2510 (such as network interface cards, modems, etc.). In embodiments, the communication interface 2510 may support wired or wireless communications, including near-field communications. These elements may be coupled to each other via a system bus 2512, which may represent one or more buses. In the case of multiple buses, they may be bridged by one or more bus bridges (not shown).

[0199] Each of these elements may perform conventional functions known in the art. In particular, system memory 2504 and mass storage device(s) 2506 may be used to store reference Figure 1 The working and permanent copies of executable code of the programming instructions of the described operating system, one or more applications and / or various software-implemented components of the mixed reality viewer 132, the mixed reality environment 134, the digital light 147, the tracker 150, the compositor 155, the user interface 157, are collectively referred to as computing logic 2522. The programming instructions that implement the computing logic 2522 may include assembly instructions supported by the processor(s) 2502 or a high-level language (e.g., C) that can be compiled into such instructions. In an embodiment, some of the computing logic may be implemented in a hardware accelerator 2575. In an embodiment, a portion of the computing logic 2522, such as a portion of the computing logic 2522 associated with the runtime environment of the compiler, may be implemented in the hardware accelerator 2575.

[0200] A permanent copy of the executable code of programming instructions or a bitstream for configuring the hardware accelerator 2575 may be placed in the permanent mass storage device(s) 2506 and / or the hardware accelerator 2575 at the factory or in the field, via a distribution medium (not shown) such as a compact disc (CD), or via a communication interface 2510 (from a distribution server (not shown)). Although for ease of understanding, the compiler and hardware accelerator that execute the generated code that incorporates the predictive computing teachings of the present disclosure to increase pipelining and / or parallel execution of nested loops are shown as being located on the same computing device, in alternative embodiments, the compiler and hardware accelerator may be located on different computing devices.

[0201] The number, functionality, and / or capacity of these elements 2510-2512 may vary depending on the intended use of the example computer device 2500, e.g., whether the example computer device 2500 is a smartphone, tablet, ultrabook, laptop, server, set-top box, game console, camera, etc. The construction of these elements 2510-2512 is otherwise known and will not be described further.

[0202] Fig.26 An example computer readable storage medium having instructions configured to implement the above referenced Figure 1 The mixed reality viewer 132, mixed reality environment 134, digital light 147, tracker 150, synthesizer 155, user interface 157 described herein may be implemented in software in whole or in part, and / or in practice as shown in FIG. 7 to FIG. Fig.24 Previously described or in conjunction with Figures 7 to Fig.24 As shown, the computer readable storage medium 2602 may include a plurality of programming instructions or executable code of a bitstream 2604. The executable code of the programming instructions (or bitstream) 2604 may be configured to enable a device, such as the computer device 2500, to perform the operations of FIG. 7 to FIG. 7 in response to the execution of the executable code / programming instructions (or the operation of the encoded hardware accelerator 2575). Fig.24 7 to Fig.24 In an alternative embodiment, the executable code / programming instructions / bitstream 2604 may be provided on a plurality of non-transitory computer-readable storage media 2602 instead. In an embodiment, the computer-readable storage media 2602 may be non-transitory. In other embodiments, the executable code / programming instructions 2604 may be encoded in a transient computer-readable medium such as a signal.

[0203] Reference again Fig.25For one embodiment, at least one of the processors 2502 may be packaged with a computer-readable storage medium having some or all of the computing logic 2522 (instead of being stored in the system memory 2504 and / or the mass storage device 2506), the computing logic 2522 being configured to practice the previously described embodiments with reference to FIGS. 7 to 8. Fig.24 All or selected operations described. For one embodiment, at least one of the processors 2502 may be packaged with a computer-readable storage medium having some or all of the computing logic 2522 to form a system-in-package (SiP). For one embodiment, at least one of the processors 2502 may be integrated on the same die with a computer-readable storage medium having some or all of the computing logic 2522. For one embodiment, at least one of the processors 2502 may be packaged with a computer-readable storage medium having some or all of the computing logic 2522 to form a system-on-chip (SoC). For at least one embodiment, a SoC may be utilized in, for example, but not limited to, a hybrid computing tablet / laptop.

[0204] It will be understood by those skilled in the art that the present disclosure is not limited to what has been specifically shown and described above. Rather, the scope of the present invention includes combinations and sub-combinations of the various features described above and modifications and variations that will occur to those skilled in the art upon reading the foregoing description. Therefore, the present disclosure is limited only by the appended claims.

Claims

1. A system comprising: Memory; processor; as well as Instructions stored in the memory and executable by the processor to: generating a 2D boundary region for a digital object to be composited onto a background image in a mixed reality view; generating a plurality of different versions of the background image at different resolutions; selecting a reference image from the plurality of different versions of the background image onto which the digital object is to be composited, the reference image having a resolution selected to vary inversely proportional to a size of the 2D border region for the digital object; as well as Values ​​for one or more digital light sources are determined based on pixels sampled from the reference image, the one or more digital light sources being used to illuminate the digital object in the mixed reality view.

2. The system of claim 1, wherein each of the multiple different versions of the background image comprises pixels that each individually represent an average of multiple pixels of the next-highest resolution images from the multiple different versions of the background image.

3. The system of claim 1 , wherein selecting one of the plurality of different versions of the background image further comprises: When the 2D boundary region for the digital object is of a first size, a background image with a lower resolution is selected than when the 2D boundary region for the digital object is of a second size, the first size being larger than the second size.

4. The system of claim 1, wherein the instructions are further executable to: The pixel resolution of the digital object is adjusted to match the resolution of the reference image.

5. The system of claim 1, wherein the instructions are further executable to: A shadow is digitally added to the digital object, the shadow being digitally added as a function of a direction of real-world lighting present in the reference image.

6. The system of claim 1, wherein the instructions are further executable to: Overlaying the 2D boundary region on the reference image and obtaining a set of samples of the colors of pixels of the reference image along the perimeter of the 2D boundary region; and Based at least in part on the set of samples, values ​​are determined for one or more digital light sources to illuminate the digital object in the mixed reality view.

7. The system of claim 6, wherein the instructions are further executable to perform at least one action selected from a group of actions comprising: shading the light reflected onto the digital object, and The intensity of light reflected onto the digital object is adjusted based on the calculated ambient light value.

8. The system of claim 7, wherein the calculated ambient light value is an average color of the set of samples.

9. A method comprising: generating a 2D boundary region for a digital object to be composited onto a background image in a mixed reality view; generating a plurality of different versions of the background image at different resolutions; selecting a reference image from the plurality of different versions of the background image onto which the digital object is to be composited, the reference image having a resolution selected to vary inversely with a size based on a dimension of the 2D boundary region for the digital object; as well as Values ​​for one or more digital light sources are determined based on pixels sampled from the reference image, the one or more digital light sources being used to illuminate the digital object in the mixed reality view.

10. The method of claim 9, wherein each of the plurality of different versions of the background image comprises pixels that each individually represent an average of a plurality of pixels of a second high resolution image from the plurality of different versions of the background image.

11. The method according to claim 10, further comprising: The pixel resolution of the digital object is adjusted to match the resolution of the reference image.

12. The method according to claim 9, further comprising: A shadow is digitally added to the digital object, the shadow being digitally added as a function of a direction of real-world lighting present in the reference image.

13. The method according to claim 9, further comprising: Overlaying the 2D boundary region on the reference image, and obtaining a set of samples of the colors of pixels of the reference image along the perimeter of the 2D boundary region; as well as Based at least in part on the set of samples, values ​​are determined for one or more digital light sources to illuminate the digital object in the mixed reality view.

14. One or more memory devices encoded with computer executable instructions for performing a computer process comprising: generating a 2D boundary region for a digital object to be composited onto a background image in a mixed reality view; generating a plurality of different versions of the background image at different resolutions; selecting a reference image from the plurality of different versions of the background image onto which the digital object is to be composited, the reference image having a resolution selected to vary inversely with a size with respect to a dimension of the 2D boundary region for the digital object; as well as Values ​​for one or more digital light sources are determined based on pixels sampled from the reference image, the one or more digital light sources being used to illuminate the digital object in the mixed reality view.

15. The one or more memory devices of claim 14, wherein each of the multiple different versions of the background image comprises pixels that each individually represent an average of multiple pixels of a second high resolution image from the multiple different versions of the background image.

16. The one or more memory devices of claim 14, wherein selecting one of the plurality of different versions of the background image further comprises: When the 2D boundary region for the digital object is of a first size, a background image with a lower resolution is selected than when the 2D boundary region for the digital object is of a second size, the first size being larger than the second size.

17. The one or more memory devices of claim 14, wherein the computer process further comprises: The pixel resolution of the digital object is adjusted to match the resolution of the reference image.

Images