Rendering using shadow information

Abstract

For a scene that includes real-world objects and computer-generated (CG) objects, the scene being characterized by scene geometry at a first time, one or more processors can generate, based on the scene geometry, shadow information for each of one or more shadows cast by a CG object of the scene on one or more real-world objects. The processors can render and display a frame of the scene from the shadow information and the scene geometry at a rendered time later than the first time. In some examples, the shadow information includes shadow factors and identifiers. Each factor can describe a light falloff effect of a shadow cast on a real-world object surface and can be incorporated into a shading atlas of the scene. Each identifier identifies a real-world object surface of the scene that is affected by the factor.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefits of the following applications: U.S. Provisional Patent Application No. 63 / 005,155, filed April 3, 2020, entitled “Method and Apparatus for Split Rendering Including Shadow Information”, and U.S. Patent Application No. 17 / 087,252, filed November 2, 2020, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] In summary, this disclosure relates to processing systems, and more specifically, to one or more techniques for graphics processing. Background Technology

[0004] Computing devices typically utilize graphics processing units (GPUs) to accelerate the rendering of graphics data for display. Such devices can include, for example, computer workstations, mobile phones such as so-called smartphones, embedded systems, personal computers, tablet computers, and video game consoles. A GPU executes a graphics processing pipeline, which includes one or more processing stages that work together to execute graphics processing commands and output frames. A central processing unit (CPU) controls the operation of the GPU by issuing one or more graphics processing commands to it. Modern CPUs are typically capable of executing multiple applications concurrently, each of which may require the GPU to utilize its resources during execution. Devices that provide content for visual presentation on a display typically include GPUs.

[0005] Typically, a device's GPU is configured to perform processes within a graphics processing pipeline. However, with the advent of wireless communication and smaller handheld devices, the need for improved graphics processing has been continuously increasing. Summary of the Invention

[0006] The following provides a brief overview of one or more aspects to offer a basic understanding of such aspects. This overview is not a comprehensive summary of all anticipated aspects, nor is it intended to identify key elements of all aspects or depict the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed descriptions that follow.

[0007] Among the various aspects of the techniques disclosed herein, example methods, computer-readable media, and apparatuses are described. In some aspects, a scene may include one or more real-world objects and one or more computer-generated (CG) objects. The scene can be characterized by scene geometry at a first-time time. In such an aspect, the technique can generate shadow information for each of one or more shadows cast by the CG objects of the scene onto one or more real-world objects in the scene, based on the scene geometry at the first-time time. The technique can then render and display frames of the scene based on: i) the shadow information, and ii) the scene geometry at a rendering time later than the first-time time.

[0008] In some of these aspects, shadow information may include: shadow factors (each shadow factor describes the light attenuation effect of a shadow cast on the surface of a real-world object, and each shadow factor is incorporated into a shading atlas of the scene) and a set of identifiers (each identifier in the set identifies a real-world object surface in the scene affected by the shadow factors). In some of these aspects, generation is performed by a first device, rendering and display are performed by a second device, and the first device communicates with the second device via a network.

[0009] In this aspect, the technique encodes a shading atlas into a video data stream prior to rendering. In this aspect, a first device sends a set of identifiers and a video stream encoded with the shading atlas to a second device via a network. The second device then receives the set of identifiers and the video stream encoded with the shading atlas from the first device and via the network, and decodes the video stream.

[0010] In some of these aspects, the video stream is a Moving Picture Experts Group (MPEG) transport stream. In some of these aspects, prior to generation, the second device determines the second device scene geometry and then sends the second device scene geometry to the first device via the network.

[0011] In one aspect, the first device receives, via a network, the scene geometry of the second device transmitted by the second device. In another aspect, the scene geometry at a given time is at least partially based on the scene geometry of the second device. In some aspects, the scene geometry of the second device includes one or more of the following: pose information of the second device, image information captured by the second device, or mesh information of the second device. In some aspects, the second device is either a video perspective device or an optical perspective device. In some aspects, generating shadow information for each of one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene includes: at least partially blocking one or more real light sources in the scene.

[0012] In some aspects, the technique can generate shadow information for each shadow in one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene, based on the scene geometry at a first time. The shadow information includes shadow factors as described above and a set of identifiers as described above. In such aspects, the first device encodes a shading atlas into a video data stream. The first device sends the set of identifiers and the video stream encoding the shading atlas to a second device via a network. In some such aspects, the video stream is a Moving Picture Experts Group (MPEG) transport stream. In some such aspects, the first device receives the scene geometry of the second device from the second device via a network prior to generation. In such aspects, the scene geometry at a first time is at least partially based on the scene geometry of the second device. In some aspects, the scene geometry of the second device includes one or more of the following: pose information of the second device, two-dimensional information captured by the second device, or mesh information of the second device. In some aspects, the second device is one of a video perspective device or an optical perspective device.

[0013] In some aspects, the technology can be implemented by a second device receiving, from a first device and via a network, a set of identifiers as described above and a video stream encoded with a shading atlas. In such aspects, the second device can perform the following operations: decoding the video stream; and then rendering and displaying frames of the scene based on: shadow information and scene geometry at a rendering time later than the first time. In some such aspects, the video stream is a Moving Picture Experts Group (MPEG) transport stream. In some such aspects, prior to reception, the second device can send its own scene geometry to the first device and via the network. In such aspects, the scene geometry at the first time is at least partially based on the sent second device scene geometry. In some such aspects, the sent second device scene geometry includes one or more of the following: pose information of the second device, two-dimensional information captured by the second device, or mesh information of the second device. In some such aspects, the second device is either a video perspective device or an optical perspective device.

[0014] Details of one or more aspects of the technology disclosed herein are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the technology disclosed herein will become apparent from the specification, drawings, and claims. Attached Figure Description

[0015] Figure 1 This is a block diagram illustrating an example content generation system based on one or more techniques according to this disclosure.

[0016] Figure 2 Example coloring atlases of one or more techniques according to this disclosure are shown.

[0017] Figure 3 An example diagram is shown illustrating communication between a client device and a server, using one or more technologies according to this disclosure.

[0018] Figure 4 Example scenarios of one or more technologies according to this disclosure are shown.

[0019] Figure 5 This is a flowchart illustrating one or more techniques based on this disclosure.

[0020] Figure 6 This is a flowchart illustrating one or more techniques based on this disclosure.

[0021] Figure 7 This is a flowchart illustrating one or more techniques based on this disclosure. Detailed Implementation

[0022] Extended Reality (XR) is an overarching term referring to a collection of technologies used in virtual reality or immersive environments, augmented or mixed reality, and other related technologies. Split rendering is a technique that enables resource-constrained mobile XR devices (clients) to display highly complex and high-fidelity computer-generated content in real time by offloading some of the necessary computation to remote rendering machines (servers) located at the network edge or in the cloud. In XR, real-world objects and computer-generated (CG) objects / content (hereinafter referred to as "CG objects") can coexist in the user's field of view and can cast shadows on each other. For example, if a CG object is blocking or occluding a real-world object from a light source, the CG object can cast a shadow onto the real-world object. However, displaying these shadows cast onto real-world objects can present numerous challenges, especially in split rendering systems or other XR systems where latency between shadows and display is significant. For example, client devices and / or servers may experience latency issues that disproportionately affect the reproduction of CG object shadows cast onto real-world objects due to the nature of the interaction between these virtual shadows and the real-world geometry that receives them. Furthermore, in the presence of these latency issues, providing an accurate and immersive depiction of shadows cast on real-world objects (as seen through XR devices) can be challenging. The examples of techniques disclosed herein address the aforementioned problems regarding shadows cast by CG objects on real-world objects in situations where there is a significant latency between content rendering and display. Moreover, the examples of techniques disclosed herein can provide an accurate depiction of shadows cast by CG objects on real-world objects.

[0023] The various aspects of the systems, apparatus, computer program products, and methods are described more fully below with reference to the accompanying drawings. However, the technology described herein can be embodied in many different forms and should not be construed as limited to any particular structure or function presented throughout this disclosure. Rather, these aspects are provided so that this disclosure will be comprehensive and complete, and will fully convey to those skilled in the art the scope of the technology described herein. Based on the teachings herein, those skilled in the art will recognize that the scope of this disclosure is intended to cover any aspect of the systems, apparatus, computer program products, and methods disclosed herein, whether that aspect is implemented independently of or in combination with other aspects of this disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth herein. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods practiced using structures, functions, or structures and functions other than or different from the aspects of this disclosure set forth herein. Any aspect disclosed herein may be embodied by one or more elements of the claims.

[0024] While various aspects are described herein, numerous variations and substitutions of these aspects fall within the scope of this disclosure. Although some potential benefits and advantages of the various aspects of the technology described herein are mentioned, the scope of this disclosure is not intended to be limited to a particular benefit, use, or objective. Rather, the aspects of this disclosure are intended to be broadly applicable to different wireless technologies, system configurations, networks, and transport protocols, some of which are illustrated by way of example in the accompanying drawings and the description below. The detailed description and accompanying drawings are illustrative only and not limiting of the technology described herein, and the claimed invention is defined by the appended claims and their equivalents.

[0025] Several aspects are given with reference to various apparatuses and methods. These apparatuses and methods are described in the specific embodiments below and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively, "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0026] For example, an element, or any part of an element, or any combination of elements, can be implemented as a "processing system," which includes one or more processors (which may also be called processing units). Examples of processors include: microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, system-on-a-chip (SoCs), baseband processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionalities described herein. One or more processors in a processing system can execute software. Whether referred to as software, firmware, middleware, microcode, hardware description languages, or others, software can be broadly interpreted as meaning instructions, instruction sets, code, code segments, program code, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc. The term can be used to refer to software. As described herein, one or more technologies can refer to an application (i.e., software) configured to perform one or more functions. In such an example, the application may be stored on memory (e.g., on-chip memory of a processor, system memory, or any other memory). The hardware described herein (such as a processor) may be configured to execute the application. For example, an application may be described as including code that, when executed by the hardware, causes the hardware to perform one or more technologies described herein. As an example, the hardware may access code from memory and execute the code accessed from memory to perform one or more technologies described herein. In some examples, components are identified in this disclosure. In such examples, a component may be hardware, software, or a combination thereof. A component may be a single component or a subcomponent of a single component.

[0027] Accordingly, in one or more examples described herein, the described functionality may be implemented in hardware, software, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media. Storage media may be any available medium accessible by a computer. By way of example, and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of computer-readable media of the types described above, or any other medium that may be used to store computer-executable code accessible by a computer in the form of instructions or data structures.

[0028] In summary, this disclosure describes techniques for having a graphics processing pipeline across one or more devices, improving the rendering of graphical content, and / or reducing latency experienced by the end viewer. For example, this disclosure describes techniques for graphics processing in devices that utilize graphics processing. Other example benefits are described throughout this disclosure.

[0029] As used herein, instances of the term "content" can refer to "graphic content," "a product of 3D graphic design" (its reproduction, i.e., "image," and vice versa), regardless of whether the term is used as an adjective, noun, or other part of speech. In some examples, as used herein, the term "graphic content" can refer to content produced by one or more processes in a graphics processing pipeline. In some examples, as used herein, the term "graphic content" can refer to content produced by a processing unit configured to perform graphics processing. In some examples, as used herein, the term "graphic content" can refer to content produced by a graphics processing unit.

[0030] In some examples, as used herein, the term "display content" can refer to content generated by a processing unit configured to perform display processing. Graphical content can be processed to become display content. For example, a graphics processing unit can output graphical content, such as frames, to a buffer (which may be referred to as a frame buffer). A display processing unit can read graphical content (such as one or more frames) from the buffer and perform one or more display processing techniques on it to generate display content. For example, a display processing unit can be configured to perform compositing on one or more rendering layers to generate frames. As another example, a display processing unit can be configured to composite, blend, or otherwise combine two or more layers into a single frame. A display processing unit can be configured to perform scaling (e.g., zooming in or out) on frames. In some examples, a frame can refer to a layer. In other examples, a frame can refer to two or more layers that have already been blended together to form a frame (i.e., a frame comprises two or more layers), and frames comprising two or more layers can subsequently be blended.

[0031] Figure 1 This is a block diagram illustrating an example system 100 configured to implement one or more technologies of this disclosure. System 100 includes device 104. Device 104 may include one or more components or circuitry for performing the various functions described herein. In some examples, one or more components of device 104 may be components of a System-on-a-Chip (SOC). Device 104 may include one or more components configured to perform one or more technologies of this disclosure. In the illustrated example, device 104 may include a processing unit 120, a content encoder / decoder 122, and system memory 124. In some aspects, device 104 may include multiple optional components, such as a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131. Reference to display 131 may refer to one or more displays 131. For example, display 131 may include a single display or multiple displays. Display 131 may include a first display and a second display. The first display may be a left-eye display, and the second display may be a right-eye display. In some examples, the first and second displays may receive different frames for presentation thereon. In other examples, the first and second displays may receive the same frames used for rendering on them. In further examples, the results of graphics processing may not be displayed on the devices; for example, the first and second displays may not receive any frames used for rendering on them. Instead, the frames or graphics processing results may be transmitted to another device. In some respects, this may be referred to as split rendering.

[0032] Processing unit 120 may include internal memory 121. Processing unit 120 may be configured to perform graphics processing, such as in a graphics processing pipeline 107. Content encoder / decoder 127 may include internal memory 123. In some examples, device 104 may include a display processor (such as display processor 127) to perform one or more display processing techniques on one or more frames generated by processing unit 120 prior to rendering by one or more displays 131. One or more displays 131 may be configured to display or otherwise render frames processed by display processor 127. In some examples, one or more displays 131 may include one or more of the following: liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, projection display device, augmented reality display device, virtual reality display device, head-mounted display, or any other type of display device.

[0033] Memory external to processing unit 120 and content encoder / decoder 122 (such as system memory 124) may be accessible to processing unit 120 and content encoder / decoder 122. For example, processing unit 120 and content encoder / decoder 122 may be configured to read from and / or write to external memory such as system memory 124. Processing unit 120 and content encoder / decoder 122 may be communicatively coupled to system memory 124 via a bus. In some examples, processing unit 120 and content encoder / decoder 122 may be communicatively coupled to each other via a bus or a different connection.

[0034] Content encoder / decoder 122 can be configured to receive graphical content from any source, such as system memory 124 and / or communication interface 126. System memory 124 can be configured to store the received encoded or decoded graphical content. Content encoder / decoder 122 can be configured to receive, for example, encoded or decoded graphical content from system memory 124 and / or communication interface 126 in the form of encoded pixel data. Content encoder / decoder 122 can be configured to encode or decode any graphical content.

[0035] Internal memory 121 or system memory 124 may include one or more volatile or non-volatile memories or storage devices. In some examples, internal memory 121 or system memory 124 may include RAM, SRAM, DRAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media or optical storage media, or any other type of memory.

[0036] According to some examples, internal memory 121 or system memory 124 may be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagating signal. However, the term "non-transitory" should not be construed as meaning that internal memory 121 or system memory 124 is immovable or that its contents are static. As one example, system memory 124 may be removed from device 104 and moved to another device. As another example, system memory 124 may be non-removable from device 104.

[0037] Processing unit 120 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), or any other processing unit that can be configured to perform graphics processing. In some examples, processing unit 120 may be integrated into the motherboard of device 104. In some examples, processing unit 120 may reside on a graphics card mounted in a port on the motherboard of device 104, or may otherwise be incorporated into a peripheral device configured to interoperate with device 104. Processing unit 120 may include one or more processors, such as one or more microprocessors, GPUs, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, processing unit 120 may store instructions for software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 121), and may execute the instructions in hardware using one or more processors to perform the technology of this disclosure. Any of the foregoing, including hardware, software, and combinations of hardware and software, can be considered as one or more processors.

[0038] The content encoder / decoder 122 can be any processing unit configured to perform content decoding. In some examples, the content encoder / decoder 122 can be integrated into the motherboard of device 104. The content encoder / decoder 122 can include one or more processors, such as one or more microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, the content encoder / decoder 122 can store instructions for software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 123), and can execute the instructions in hardware using one or more processors to perform the technology of this disclosure. Any of the foregoing, including hardware, software, combinations of hardware and software, etc., can be considered as one or more processors.

[0039] In some aspects, system 100 may include an optional communication interface 126. Communication interface 126 may include a receiver 128 and a transmitter 130. Receiver 128 may be configured to perform any of the receiving functions described herein with respect to device 104. Additionally, receiver 128 may be configured to receive information from another device (e.g., eye or head position information, rendering commands, or location information). Transmitter 130 may be configured to perform any of the transmitting functions described herein with respect to device 104. For example, transmitter 130 may be configured to send information to another device that may include a request for content. Receiver 128 and transmitter 130 may be combined into transceiver 132. In such an example, transceiver 132 may be configured to perform any of the receiving and / or transmitting functions described herein with respect to device 104.

[0040] Refer again Figure 1 In some aspects, a scene may include one or more real-world objects and one or more computer-generated (CG) objects. The scene may be characterized by scene geometry at a first-time time. In such an aspect, the graphics processing pipeline 107 may include a determining component 198 configured to generate shadow information for each of one or more shadows cast by the CG objects of the scene onto one or more real-world objects in the scene, based on the scene geometry at the first-time time. The graphics processing pipeline 107 can then render and display frames of the scene based on: i) the shadow information, and ii) the scene geometry at a rendering time later than the first-time time.

[0041] In some of these aspects, shadow information may include: shadow factors (each shadow factor describes the light attenuation effect of a shadow cast on the surface of a real-world object, and each shadow factor is incorporated into a shading atlas of the scene) and a set of identifiers (each identifier in this set identifies a real-world object surface in the scene affected by the shadow factors). In some of these aspects, generation is performed by a first device, rendering and display are performed by a second device, and the first device communicates with the second device via a network. In some of these aspects, prior to rendering, the graphics processing pipeline 107 encodes the shading atlas into a video data stream. In some of these aspects, the first device sends the identifier set and the video stream encoded with the shading atlas to the second device via the network. The second device then receives the identifier set and the video stream encoded with the shading atlas from the first device and via the network, and decodes the video stream.

[0042] In some aspects, the video stream is a Moving Picture Experts Group (MPEG) transport stream. In some aspects, prior to generation, the second device determines its own scene geometry and then transmits it to the first device via a network. In some aspects, the first device receives the transmitted second-device scene geometry from the second device via a network. In some aspects, the scene geometry at a given time is at least partially based on the second-device scene geometry. In some aspects, the second-device scene geometry includes one or more of the following: pose information of the second device, image information captured by the second device, or mesh information of the second device. In some aspects, the second device is either a video perspective device or an optical perspective device. In some aspects, the shadow information for generating each shadow in one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene includes at least partially blocking one or more real light sources in the scene.

[0043] As described herein, a device such as device 104 can refer to any one or more devices, apparatuses, or systems configured to perform one or more of the technologies described herein. For example, a device can be a server, base station, user equipment, client device, station, access point, computer (e.g., personal computer, desktop computer, laptop computer, tablet computer, computer workstation, or mainframe computer), end product, apparatus, telephone, smartphone, server, video game platform or console, handheld device (e.g., portable video game device or personal digital assistant (PDA)), wearable computing device (e.g., smartwatch, augmented reality device, or virtual reality device), non-wearable device, display or display device, television, set-top box, intermediate network device, digital media player, video streaming device, content streaming device, in-vehicle computer, any mobile device, any device configured to generate graphical content, or any device configured to perform one or more of the technologies described herein. The processes described herein may be described as being performed by a specific component (e.g., GPU), but in further embodiments, they may be performed using other components (e.g., CPU) consistent with the disclosed embodiments.

[0044] A GPU can process various types of data or data packets within its pipeline. For example, in some aspects, a GPU can process two types of data or data packets, such as context register packets and draw call data. A context register packet can be a collection of global state information that manages how the graphics context will be processed, such as information about global registers, shaders, or constant data. For example, a context register packet may include information about color formats. In some aspects of a context register packet, there may be bits indicating which workload belongs to the context register. Furthermore, there may be multiple functions or programs running simultaneously and / or in parallel. For example, a function or program may describe a specific operation, such as a color mode or color format. Therefore, context registers can define multiple states of the GPU.

[0045] Context states can be used to determine how individual processing units (e.g., vertex extractors (VFDs), vertex shaders (VSs), shader processors, or geometry processors) operate, and / or in which mode a processing unit operates. For this purpose, the GPU can use context registers and programming data. In some aspects, the GPU can generate workloads, such as vertex or pixel workloads, in the pipeline based on context register definitions of modes or states. Certain processing units (e.g., VFDs) can use these states to determine certain functions, such as how to assemble vertices. Because these modes or states can change, the GPU may need to modify the corresponding context. Furthermore, the workload corresponding to a mode or state can follow a constantly changing mode or state.

[0046] GPUs can render images in a variety of different ways. In some cases, GPUs can render images using either rendering or tiled rendering. In tiled rendering GPUs, an image can be segmented or separated into different segments or tiles. After the image is segmented, each segment or tile can be rendered individually. Tiled rendering GPUs can divide computer graphics images into a grid format, allowing each part of the grid (e.g., a tile) to be rendered individually. In some aspects, during the binning pass, an image can be divided into different bins or tiles. Furthermore, in the binning pass, different primitives can be shaded in certain bins, for example, using draw calls. In some aspects, during the binning pass, a visibility stream can be constructed, where visible primitives or draw calls can be identified.

[0047] In some aspects of rendering, multiple processing stages or pathways can exist. For example, rendering can be performed in two pathways, such as a visibility pathway and a rendering pathway. During the visibility pathway, the GPU can input a rendering workload, record the positions of primitives or triangles, and then determine which primitives or triangles fall into which part of the frame. In some aspects of the visibility pathway, the GPU can also identify or mark the visibility of each primitive or triangle in the visibility stream. During the rendering pathway, the GPU can input a visibility stream and process one part of the frame at a time. In some aspects, the visibility stream can be analyzed to determine which primitives are visible or invisible. Thus, visible primitives can be processed. By doing so, the GPU on the server can reduce the unnecessary workload of processing or rendering invisible primitives.

[0048] In some respects, rendering can be performed in multiple locations and / or on multiple devices, for example, to divide the rendering workload across different devices and / or to address latency issues. For instance, rendering can be split between server and client devices, which can be referred to as "split rendering." In some cases, split rendering can be a method for bringing content to a user device or head-mounted display (HMD), where a portion of the graphics processing can be performed outside of that device or HMD (e.g., at the server).

[0049] Split rendering can be performed for a variety of different types of applications, such as extended reality (XR) applications. In XR applications, the content displayed on the user's device can correspond to artificial or computer-generated (CG) content, such as content rendered on a server or user's device. In XR content, a portion of the content displayed on the user's device can correspond to real-world content (e.g., real-world objects), and a portion of the content can be CG content. Furthermore, CG content and real-world content can be displayed in optical or video perspective devices, allowing users to view real-world objects and CG content simultaneously. In some respects, CG content can be referred to as artificial content, animated content, or augmented content, or vice versa.

[0050] Examples of the techniques disclosed herein can be considered for various modalities of object-space rendering representation, such as a vector streaming (VS) architecture. In a VS architecture, server 310 can stream object-space representations (using shading atlases, shadow material messages, and geometric descriptions) for real-world and CG objects. For example, in a vector streaming architecture, the server can generate shading atlas information, such as representations of textures in the scene, and / or geometric information (e.g., information about the geometry of objects corresponding to these textures). For example, the server in a vector streaming system can also include information about the mapping between geometric primitives (e.g., triangles) and shading atlas primitives (e.g., tiles or blocks). By providing this information, the server can provide the client with texture coordinates in the atlas and the target screen position in the display frame for compositing the final display frame through texture mapping.

[0051] In XR applications, CG objects may occlude other objects (both real-world and CG) or cast shadows on those other objects from a vantage point on the user device 330. This overlap between real-world and CG content that produces the aforementioned occlusion / shadowing is one reason why CG and real-world content can be seamlessly blended within XR.

[0052] Besides occlusion, real-world and CG content can also cast shadows from one to the other, and realistic shadow casting is another important aspect of creating a seamless XR experience. For example, if a CG object is situated between a light source and a real-world object, the appearance of the real-world object should be appropriately modified to account for the corresponding loss of brightness in the shadows cast by the CG object. It should be noted that the light source can be either a real-world light source or a CG light source, as both can appear seamlessly to XR application users. Additionally, it should be noted that realistic-looking shadows are obtained as a combination of the shape of the shadow caster's outline as seen by an obstructed light source and the geometry of the shadow receiving object. In the case considered here, the shadow caster is a CG object, and the shadow receiving object of interest is a real-world object whose illumination from a particular light source is blocked or partially blocked by the shadow caster. One challenge in split rendering systems may be how to make the shadow effects cast from a CG object onto a real-world object as latency-insensitive as possible, given that the shape, outline, and "intensity" of the shadow are so dependent on the geometry and relative position of both the shadow caster and the shadow receiver, as well as the obstructed light source.

[0053] Figure 2 A color atlas 200 illustrating one or more techniques according to this disclosure is shown. For example... Figure 2 As shown, the shading atlas 200 demonstrates an efficient way to store textures in object space rather than image space. Figure 2 It is also shown that different portions of the shading atlas 200 are shaded at different resolutions, for example, depending on the distance from the camera. Furthermore, dark gray portions (e.g., shadow 210 or portions along the right edge of the shading atlas 200) can represent unassigned portions of the shading atlas. In some cases, the shading atlas 200 can be efficiently encoded based on high temporal coherence. For example, a block in the shading atlas 200 can represent the same physical surface in the virtual world. In some aspects, blocks in the shading atlas 200 can remain in the same location, as long as they are potentially visible and / or occupy similar areas in screen space.

[0054] In some aspects, color atlases can be used to send coloring information from a server to a client device. Color atlases can also be video encoded. In some cases, even if the encoded color atlas conforms to a standard Moving Picture Experts Group (MPEG) stream, at least some of the attributes and / or information included in the color atlas can be unique for color atlas transmission. For example, in cases where the color atlas being encoded changes compared to a previous frame (e.g., "atlas reset"), IDR frames can be inserted.

[0055] Figure 3A vector streaming architecture 300 is illustrated, comprising an information transmission protocol between server 310 and client device 330 (hereinafter referred to as "client 330"). Architecture 300 includes server 310, network 320, and client 330. Figure 3 As shown, server 310 includes a memory management and visibility phase 312, a shading phase 314, and an encoding phase 316. Client device 330 may include a decoding phase 332, a rendering and display phase 334, and a pose estimation phase 336.

[0056] like Figure 3 As shown, the memory management and visibility phase 312 can determine the visible and potentially visible scene primitives that need to be sent to the client. Based on this information, the server 310 determines which polygons need to be shaded and where they will be fitted in the atlas. This information is provided to the shading phase 314. The shading phase 314 can determine the relevant textures within the shading atlas 343 to be sent to the client 330 for the current frame. The shading atlas 343 can contain all potentially visible surfaces in the XR scene that are packaged as shading textures. The encoding phase 316 can be responsible for encoding a series of shading atlas frames into an MPEG stream 344. The MPEG stream 344 (including the encoded shading atlas 343) can be transmitted between the server 310 and the client 330 via the network 320. The client 330 decodes the stream received via the network 320 in the decoding phase 332. After decoding, the rendering and display phase 334 can use the shading atlas 348 and auxiliary information 349 along with the most recent pose information 352, for example, to render and display the left and right eye buffers.

[0057] In some aspects of split rendering, decisions related to object geometry can be resolved during the visibility pathway (e.g., within the game engine). In a split-render XR system, the game engine may reside on server 310. The visibility pathway (e.g., in memory management and visibility phase 312) may consider the poses of all relevant objects (e.g., real-world objects and CG objects in the current camera frustum (depending on the camera / client 330 pose)) and / or determine which triangles are visible or potentially visible from the current camera pose.

[0058] Split rendering architectures can also introduce latency when delivering rendered content to the client 330 display. In some aspects, this latency, measured between determining an instance of client pose and displaying an instance of CG content corresponding to that client pose on the client display, may be higher when shading occurs entirely or partially on the server 310 than when rendering is performed on the client device 330. However, in such a setup where shading is performed on the server, more complex XR graphics can also be enabled. For example, there may always be a certain amount of latency in a split rendering architecture, and different split rendering systems may incorporate different mechanisms to handle or compensate for latency.

[0059] In a vector streaming system (e.g., a split rendering system based on object-space rendering representation (SRS OSRR)), server 310 can stream the appearance of various objects in object space, as well as the geometry of these objects. Object surfaces shaded on the server side can include currently visible surfaces given the latest client pose, as well as potentially visible surfaces that may be revealed after slight movements of the client device near the original pose. In later stages of processing (e.g., before display), client 330 can rasterize the geometry of visible objects and / or texture map the corresponding textures based on recent local pose information. This variant of pre-display processing inherently has no residual pose-to-display latency because the most recently available pose is used in a timely manner to re-render the content for display. This variant of pre-display processing can avoid distorting the shape of objects and preserve the screen-space displacement of various elements in the mixed reality scene, as they will naturally emerge based on the depth of the mixed scene content and relative camera motion. Another aspect of the same property is that occluded / occluded surfaces can appear as expected; in other words, some surfaces may be unoccluded due to potential camera movement, while others may be more occluded than they are during server rendering. It should be noted that any reference to "camera" in this document may refer to multiple cameras, virtual cameras, client devices, displays, multiple displays, and / or HMDs.

[0060] Vector streaming systems, or SRS OSRR, can also stream different types of information. For example, in vector streaming architecture 300, server 310 can generate: 1) shading atlas information 343, such as a representation of textures in the scene; and 2) auxiliary information 342, such as information about the geometry / mesh of objects and the potential visible polygons in those meshes seen in frames to be sent to client 330, and any additional information describing the mapping between the potential visible polygons and the textures shaded in the shading atlas. In some aspects, the shading atlas information can be sent as a time-evolved texture and encoded as video, for example, through encoding stage 316. In some aspects, the auxiliary information 342 can partially represent the mesh or a portion of the mesh of a CG object and can be transmitted raw or encoded using mesh compression methods. Client device 330 can receive both the shading atlas 343 and the auxiliary information 342 corresponding to pose T0 or a server frame, which can decode 332 and use it to render a final eye buffer representation of the scene using the most recent pose T1. Since pose T1 is more recent and closer to pose T0, the system described is less constrained by latency compared to other split rendering architectures.

[0061] Client device 330 can use a closer pose T1 to rasterize the shading atlas 348 and auxiliary information 349 (i.e., convert the information into pixels that can be displayed on client 330). Additionally, in some aspects, server 310 can receive pose information 352 streamed from client device 330 and then perform visibility 312 and shading 314 calculations based on the received pose information 352, or multiple received poses, or a time-advance / predicted version of the received poses. Visibility calculation 312 can affect which meshes (geometry) or portions of meshes can be sent to client device 330, and shading calculation 314 can include shading surfaces corresponding to meshes or portions of meshes that need to be updated in the current frame, which can also be sent to client 330. Examples of the techniques disclosed herein can also assume that client 330 has information about these meshes, allowing geometry to be streamed to client device 330 without real-time transmission. For example, server 310 can calculate the shading on objects, for example, the shading can be changed based on the advantage of a camera (e.g., a virtual camera in a game engine and / or game state), and client 330 may already know the scene geometry information, so it only needs to incorporate the newly received shading information.

[0062] In some aspects, real-world objects in a mixed reality scene can be represented on the client device 330 as a grid describing the object's position, shape, and size. Furthermore, this grid geometric representation can be determined or known at the server 310 and / or the client 330. Real-world geometric information is required to synthesize a fully immersive mixed reality scene. In some cases, knowing the geometry of real-world objects allows the XR system to "occlude" portions of CG content intended to appear behind the real-world objects. This information can also be used to calculate the lighting produced when CG content is tinted and the shadow effects cast by the CG content onto real-world objects. These are some examples where real-world geometry can be useful to an immersive XR system.

[0063] If real-world geometry is known at both the client 330 and the server 310, some information in the data used to represent the real-world content may not need to be sent by the server 310. Furthermore, from a latency perspective, having real-world meshing capabilities on the client device 330 has its advantages. If the real-world scene is not entirely static (i.e., it contains moving or deformable objects), the latency involved in a system that fully meshes the world on the server 310 and brings the mesh information to the rasterization at the client 330 may begin to take effect on the user experience. Client devices 330 with on-device meshing capabilities can have an advantage because their real-world representation components can induce lower latency and therefore correspond more closely to the actual perspective scene.

[0064] Client 330 can also perform a coarse real-world mesh and send that information to server 310. Server 310 can then perform a bundle adjustment, comparing real-world observations of real objects made by client 330 at many different points in time and refining that information into a single, compact, refined geometry for future representations of the real-world objects. Therefore, real-world content refinement can occur on either client 330 or server 310. In some cases, client 330 can utilize additional pathways to render occluded objects unknown to server 310, for example, using the latest real-world object mesh and pose, understanding that in some cases the texture information corresponding to these objects may be somewhat incomplete before receiving updates from server 310.

[0065] As noted above, segmented rendering for XR can include methods for bringing content to a headset or HMD (i.e., client device 330), wherein a portion of the graphics processing is performed outside the headset (e.g., on server 310). Furthermore, mixed reality XR can include a portion of the screen corresponding to real-world content and displayed in an optical or video perspective device. Therefore, in XR, the object viewed in the headset or HMD can be a mixture of real-world content and CG content (e.g., animated content or enhancements).

[0066] To make CG objects and the shadows they cast appear realistic, several different steps or processes can be performed. For example, light or shadow from a CG object can be realistically projected onto a real-world object, or vice versa. Therefore, real-world objects and CG objects can cast shadows on each other. For instance, if a CG object is blocking or occluding a real-world object from a light source in object space, the CG object can cast shadows on the real-world object, and thus may require modifications to the appearance of the real-world object when presented to the viewer. In a split rendering system, some of these calculations can be offloaded to the server.

[0067] However, the aforementioned shadows cast by CG objects onto real-world objects can present several challenges. For example, before the shadow information is transmitted to the second device (e.g., client 330), the split rendering system generates the shadow information on the first device (e.g., server 310). Client 330 receives, decodes, and interprets this information and modifies its display values ​​accordingly. Therefore, the signal carrying the shadow information (which should affect the real object surface when displayed) is subject to potentially high system latency, while the real-world object itself appears in the second device (optical perspective device) with no or negligible latency (video perspective device). Consequently, the shadow information (from the CG object) may be subject to hysteresis, which may manifest, for example, in inaccurate display positions, shapes, and areas affected by the shadow. Assuming that the object surface (real-world or CG) intended to be the recipient of the shadow may "move" on the monitor with little or no latency compared to the movement of the display device, providing an accurate depiction of shadows cast on both real-world and augmented objects with the same fidelity and little or no artifacts can be challenging.

[0068] The techniques disclosed herein can address the aforementioned problem regarding shadows cast by CG objects onto real-world objects. These techniques can generate shadow information on a first device (e.g., server 310) and utilize that shadow information on a second device (e.g., client 330) in such a way that, to the end user operating the second device, the shadows of the CG objects on the real-world objects appear to be generated locally with minimal to no latency. Furthermore, the shadow information can be provided to the second device not only for real-world object surfaces visible when the server begins the rendering process, but also for surfaces that may become visible in the near future as the second device moves around. This helps the second device characterize shadows on surfaces that are invisible from the second device's perspective when the first device is generating shadow information, but become visible or are revealed / removed between that time and the time when the shadow information is used on the second device to change the appearance of the scene on the display. Moreover, the techniques disclosed herein can provide an accurate depiction of shadows cast by CG objects onto real-world objects. By doing so, these techniques can help achieve more realistic and immersive XR experiences.

[0069] In some examples, the techniques disclosed in this paper can be applied to vector streaming pipeline architectures (such as...) Figure 3 The aforementioned shadow information is processed in the vector streaming pipeline architecture 300 shown herein. For example, geometric information about real-world surfaces (e.g., via a spatial map) may be available at server 310, generated from online or offline meshing. This auxiliary information 342 may also be available on client device 330, for example, if client 330 performs surface meshing. Furthermore, light estimation in the scene may be performed on client 330 or at server 310, and light source information may be available at server 310 during the shading computation. Some examples of the techniques disclosed herein may be implemented using client 330 as a headset or HMD, which utilizes a video perspective display or optical perspective display to display objects and surfaces rendered per client 330 pose.

[0070] To accurately simulate shadows cast by CG objects onto real-world objects, the geometric information of the real-world objects can be determined. In some cases, this can be achieved through a combination of computer vision techniques, 3D reconstruction, and / or meshing. In one scenario, meshing can be performed in real-time on client 330, and the reconstructed geometry can be transmitted to server 310. In another scenario, client 330 can capture images sequentially and send them to server 310, where server 310 can perform 3D reconstruction and / or meshing algorithms to extract 3D mesh information about objects visible in the real-world scene. For example, in some examples of the techniques disclosed herein, a real-world 3D mesh can be determined. Accordingly, the geometry of the real-world object can be available on client 330 and sent to server 310, or vice versa. As more observations of the same real-world or CG object become available, this geometric representation can be adjusted and become more accurate over time. By doing so, the geometric information about the object can become increasingly accurate over time.

[0071] Examples of the techniques disclosed herein may include several different steps or processes for processing shadows cast by CG objects in split rendering. For example, examples of the techniques disclosed herein may, for instance, determine which real-world surfaces are visible via server 310. Starting with these visible surfaces, server 310 may determine which visible or potentially visible real-world surfaces are receiving shadows from CG content. Additionally, server 310 may calculate shadow factors (e.g., with values ​​from 0 to 1) for these surfaces, the shadow factor being used to quantify how much incident light is attenuated / blocked before reaching the surface. Thereafter, server 310 may store shadow information including the shadow factors. For example, a value of 0 may correspond to no shadow, while a value of 1 may correspond to maximum light attenuation on a video perspective pixel. Furthermore, server 310 may stream information about which geometry rendered on client device 330 requires the corresponding texture to be treated as a shadow material, for example, by rendering the corresponding surface in a shading atlas as transparent.

[0072] Furthermore, the client device 330 can render CG surfaces, as well as essentially real-world surfaces, thereby converting from object space to display frame space. The client device 330 can sample shadow factors and / or utilize the meshes of CG objects and real-world objects. In some cases, the meshes of real-world objects can be estimated on the client device 330 or on the server. It can be assumed that these meshes of the real-world objects are known to both the server and the client device 330. For example, it can be assumed that the geometry, size, and / or shape of the meshes are known to both the server and the client device 330.

[0073] In some examples, the techniques disclosed herein can calculate shadow information for CG objects used to cast shadows on real-world objects. The shadow information calculated on server 310 covers both visible and potentially visible real-world objects, even if not all objects are simultaneously visible on the display of client device 330. Additionally, as mentioned above, the XR applications of this paper can access geometric information for all objects in a mixed reality scene: both real-world objects and CG objects used to describe shadows or occlusions.

[0074] In some cases, when calculating shadow information about a shadow cast by a CG object onto a real-world object, components of a shading atlas can be used to store the corresponding shadow mask information (e.g., as a shadow factor). Conversely, when calculating information about the shadow cast on a CG object, this can be automatically included in the shading atlas's regular surface. Therefore, when CG content casts a shadow on a real-world object, or when a real-world object casts a shadow on CG content, the examples of techniques disclosed herein can determine the corresponding information to be included in the shading atlas. However, when a real-world object casts a shadow on another real-world object, this disclosure may not make any determination or use of information from the shading atlas, as that information may already be included in the visible scene.

[0075] In some aspects, if a CG object is casting a shadow on a real-world object, that portion of the real-world object can be modulated to reduce or suppress the lighting perceived by the viewer on that portion of the real-world object through the monitor. This shadow information can be included in a shadow mask or shadow texture and consumed as a shadow material on the client. Accordingly, when a CG object casts a shadow on a real-world object, the shadow material can be applied to the real-world object during rendering at 330 on the client. By doing so, the effects of real-world or CG light sources can be simulated uniformly on the real-world or CG object on the server and transmitted and applied accordingly to the client.

[0076] As noted above, the examples of techniques disclosed herein can determine the amount of light received and reflected by a CG object or a real-world object. Such examples can also determine the portions of a CG object or real-world object that will be affected by this light. The corresponding shadow information for the object can then be calculated in a simplified manner and stored as a shadow factor in a shadow mask or shadow texture. If it may be necessary to use the shadows cast by the CG object to characterize a real-world object, the client 330 can use the shadow factor from the shadow mask included in the shading atlas to render a shadow material covering the corresponding object's mesh.

[0077] In some respects, the client device 330 can rasterize the mesh and texture map the surface as if shadows did not exist. This may be the case if the system lacks the ability to characterize shadows on real-world objects or to estimate the geometry of real-world objects and surfaces. Additionally, some real-world object surfaces may not be included in the shading atlas because these objects may not be recipients of any shadows and therefore may not require any draw calls. For example, some real-world objects may not be receiving any shadows from CG objects or may be occluded by CG objects; they should appear when not viewed using a headset or HMD, so the atlas may not need to include information about these real-world objects. As noted above, the shading atlas may include various shadow information for any shadows cast on the augmented content and / or any shadows cast on the augmented content by real-world objects. It should be noted that the shading atlas contains information about the shading of texels, not just shadows. Shadows can affect how texels appear. However, texel shading can depend on several other factors. In addition to simple shading of texels, the shading atlas can also be used to carry shadow masking information for certain real-world objects. The latter is not the primary function of the atlas, but it can be used for this purpose, as in the examples of the techniques disclosed herein. Alternatively, the shadow mask can be sent to the client in a separate texture.

[0078] As previously noted, shadow information can be included in a shadow mask, which can be inserted into a shading atlas. In addition to describing the shadow mask information of real-world light sources partially occluded by the CG object from the perspective of the real-world object, in some cases, the client 330 also needs to receive a list of tile identifiers describing the surface of the real-world object affected by these shadows. Upon receiving this information, the appearance of the real-world object affected by CG shadows can be modified on the client by additionally rendering the information described by the shadow mask and tile identifiers, thus making the real-world object appear more realistic.

[0079] As noted above, the examples of techniques disclosed herein can modify the perceived lighting of a real-world object (e.g., as seen through headphones) based on shadow information when a shadow is projected from an animated object onto a real-world object. In video perspective devices, the shadow / color corresponding to the real-world material receiving the shadow can be appropriately modified / darkened to faithfully depict the effect of shadow projection. In optical perspective devices, the amount of light reaching the eye from a specific direction (given by the screen projection of the shadow-receiving surface) needs to be attenuated by the headphone medium.

[0080] Examples of techniques disclosed herein can also address several of the aforementioned shadow problems. For example, at high latency, shadows may suffer from geometric distortion if they are not described in a way that remains accurate after a specific amount of headphone displacement from the location used to shading the scene. As noted herein, examples of techniques disclosed herein can address and improve upon problems concerning shadow characterization, for example, in a traditional, bulletin board representation. Furthermore, shadows may exhibit jitter and / or fail to adhere well to object surfaces when they lack an accurate geometric description accompanying them to the surfaces they need to adhere to. Additionally, shadows or other indirect lighting contributions, besides color and alpha video used for XR content, may utilize an increased amount of bandwidth. As noted herein, split rendering methods according to examples of techniques disclosed herein can address each of these problems, such as by reducing sensitivity to latency and / or improving the accuracy and seamlessness of shadows on real-world objects (even when projected by non-static CG content). To do this, examples of techniques disclosed herein can send shading information corresponding to visible and potentially visible real-world or augmented surfaces in the shading atlas, which can be optimized for efficient storage and do so in a way that remains accurate under a specific amount of headphone displacement.

[0081] Figure 4 Example images or scenes 400 illustrating one or more examples of the techniques disclosed herein are shown. Scene 400 includes a CG surface or object 460, a real-world surface or object 470, a real-world surface or object 480, a real-world object (sphere) 485, and a light source 490. Additionally, scene 400 includes a shadow 462 cast by the CG object 460 based on the light source 490. Figure 4 This shows a CG object 460 (e.g., a person) positioned between a light source 490 and real-world objects 480 and 484. Figure 4 As shown, CG content 460 projects a shadow (e.g., shadow 462) onto both real-world objects 480 and 485, wherein the shape and extent of the shadow depend on the position and shape (geometry) of the light source 490 and CG content 460, as well as the geometry of the real-world surfaces 480 and 485 that receive the shadow.

[0082] See below for reference Figure 4As described herein, examples of the techniques disclosed herein can accurately depict when CG content 460 is located between a light source 490 (real world or CG) and real-world objects (ground 480 and sphere 485). Therefore, examples of the techniques disclosed herein can accurately depict when shadows are cast from CG content onto real-world objects. Examples of the techniques disclosed herein (e.g., server 310 and client 330 of this document) can perform multiple different steps or processes to accurately depict shadows cast by CG content onto the real world, even with a significant time delay between the moment shadow information is calculated and the moment client 330 uses the shadow information.

[0083] refer to Figure 5 Furthermore, referring to the preceding figures for context, a flowchart of an example method 500 according to one or more techniques in accordance with this disclosure is shown.

[0084] Consider a scene comprising one or more real-world objects and one or more computer-generated (CG) objects. The scene is characterized by first-time scene geometry. Method 500 generates shadow information (box 510) based on the first-time scene geometry for each of one or more shadows cast by the CG objects of the scene onto one or more real-world objects in the scene. In a continuing example, consider scene 400, which includes CG object 460, real-world object 470, real-world object 480, real-world sphere 485, and light source 490. The scene is characterized by auxiliary information 342 regarding the scene geometry of the objects and potentially visible polygons in these meshes, as seen in a frame. Method 500 generates shadow information for shadow 462 cast by CG object 460 that occludes light source 490 from the surfaces of real-world objects floor 480 and sphere 485. In some examples, the method generates shadow information for each of one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene by at least partially blocking one or more real light sources in the scene.

[0085] In some examples, a method or apparatus for graphics processing is provided. The apparatus may be one or more of the following: a server, a client device, a CPU, a GPU, or some other processor capable of performing graphics processing. In one aspect, the apparatus may be a processing unit 120 within device 104 or some other hardware within device 104 or another device. The apparatus may include: a unit for generating shadow information for each of one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene, based on scene geometry at a first time.

[0086] return Figure 5Method 500 renders and displays frames of the scene based on: i) shadow information, and ii) scene geometry at a later rendering time than the first time (box 520). In a continuing example, method 500 renders and displays frames of scene 400 based on: shadow information for shadow 462 cast by CG object 460 that blocks light source 490 from the surfaces of real-world objects floor 480 and sphere 485, and scene geometry that has changed since the shadow information was generated.

[0087] In some examples, the apparatus may be a processing unit 120 within device 104, or some other hardware within device 104 or another device. The apparatus may include units for rendering and displaying frames of the scene according to: i) shadow information, and ii) scene geometry at a rendering time later than the first time.

[0088] refer to Figure 6 Referring further to the preceding figures for context, a flowchart of an example method 600 according to one or more techniques of this disclosure is shown. In such method 600, shadow information includes shadow factors and a set of identifiers. Each shadow factor describes the light attenuation effect of a shadow cast on the surface of a real-world object. Each shadow factor is incorporated into a shading atlas of the scene. Each identifier in the set identifies a real-world object surface in the scene affected by the shadow factor. In such method 600, the generation of box 510 is performed by a first device; and the rendering and display of box 520 are performed by a second device. The first device communicates with the second device via a network. In some examples, the second device is either a video perspective device or an optical perspective device.

[0089] In such a method 600, the first device encodes the shading atlas into the video data stream prior to rendering (box 630). In a continuing example, the first device is server 310. Server 310 encodes the atlas 343 into the MPEG stream 344 in encoding stage 316. In some examples, the apparatus may include units for encoding the shading atlas into the video data stream prior to rendering.

[0090] The first device sends a set of identifiers and a video stream encoded with a color atlas to the second device via a network (box 640). In a continuing example, server 310 sends a set of identifiers and an MPEG stream encoded with a color atlas 343 to client 330 via network 320. In some examples, the apparatus may include units for sending a set of identifiers and a video stream encoded with a color atlas to the second device via a network.

[0091] The second device receives a set of identifiers and a video stream encoded with a color atlas from the first device and via a network (box 650). In a continuing example, client 330 receives an MPEG stream 344 and a set of identifiers sent by server 310 via network 320. In some examples, the apparatus may include units for receiving a set of identifiers and a video stream encoded with a color atlas from the first device and via a network.

[0092] The second device decodes the video stream (box 660). In a continuing example, client 330 decodes MPEG stream 344 to reveal color atlas 348 and auxiliary information 349 including a set of identifiers. In some examples, the device may include a unit for decoding the video stream.

[0093] In the continued example, client 330 then renders and displays frames of scene 400 based on: i) shadow information from the decoded shading atlas 348 and identifier set, and ii) scene geometry, such as geometry and pose estimates 336 at the time of rendering. The rendering time is later than the initial time at which scene geometry is used to generate shadow information at server 310, primarily due to latency in processing and communication from server 310 to client 330. More current scene geometry and pose estimates 336 at the time of rendering and display at client 330 will mitigate the impact of this latency.

[0094] refer to Figure 7 And continuing to refer to the preceding figures for context, a flowchart of an example method 700 according to one or more techniques of this disclosure is shown. In such method 700, blocks 510, 520 and 630-660 are combined as described above. Figure 5 and Figure 6 The process is performed as described. In such a method 700, prior to generation, the second device determines the second device scene geometry (box 770). In a continuing example, the geometry and pose estimation 336 of the client 330 determines the second device geometry information. The second device scene geometry may include one or more of the following: pose information of the second device, image information captured by the second device, or mesh information of the second device. In some examples, the apparatus may include units for determining the second device scene geometry via the second device.

[0095] The second device sends its scene geometry to the first device and over the network (box 780). In a continuing example, client 330 sends the second device geometry to server 310 via network 320. In some examples, the apparatus may include units for sending the second device scene geometry to the first device and over the network.

[0096] The first device receives the second device scene geometry transmitted via a network from the second device (box 790). In a continuing example, the server 310 receives, via network 320, the client 330 scene geometry determined by the client's geometry and pose estimation stage 336. The received scene geometry is the first-time scene geometry that the server will use to generate shadow information, as described above in conjunction with box 510. In some examples, the apparatus may include units for receiving the transmitted second device scene geometry from the second device via a network.

[0097] As a supplementary explanation, in Example 1, the techniques disclosed herein can be methods for graphics processing, one or more apparatuses, or computer-readable media storing computer-executable code. In such an example, for a scene comprising one or more real-world objects and one or more computer-generated (CG) objects, the scene being characterized by scene geometry at a first time, the method can generate shadow information based on the scene geometry at the first time for each of one or more shadows cast by the CG objects of the scene onto one or more real-world objects in the scene. The example can then render and display frames of the scene according to: i) the shadow information, and ii) the scene geometry at a time later than the first time of rendering.

[0098] Example 2 includes Example 1, wherein the shadow information includes: shadow factors, each shadow factor i) describing the light attenuation effect of a shadow cast on a real-world object surface and ii) being incorporated into a shading atlas of the scene; and a set of identifiers, each identifier in the set identifying a real-world object surface of the scene affected by the shadow factors. Example 3 includes any one of Examples 1-2, wherein: the generation is performed by a first device, the rendering and the display are performed by a second device, and the first device communicates with the second device via a network. Such examples further include: encoding the shading atlas into a video data stream prior to the rendering; sending the identifier set and the video stream encoded with the shading atlas to the second device by the first device and via the network; receiving the identifier set and the video stream encoded with the shading atlas from the first device and via the network by the second device; and decoding the video stream. Example 4 includes any one of Examples 1-2, wherein the video stream is a Moving Picture Experts Group (MPEG) transport stream. Example 5 includes any one of Examples 1-4, and further, prior to the generation: the second device determines the second device scene geometry; the second device sends the second device scene geometry to the first device via the network; and the first device receives the sent second device scene geometry from the second device via the network. In such an example, the scene geometry at the first time is at least partially based on the second device scene geometry. Example 6 includes any one of Examples 1-5, wherein the second device scene geometry includes one or more of the following: pose information of the second device, image information captured by the second device, or mesh information of the second device. Example 7 includes any one of Examples 1-6, wherein the second device is a video perspective device or an optical perspective device. Example 8 includes any one of Examples 1-7, wherein generating shadow information for each of one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene includes: at least partially blocking one or more real light sources in the scene.

[0099] In Example 9, the techniques disclosed herein may be methods for graphics processing, one or more apparatuses, or computer-readable media storing computer-executable code. In such an example, the scene includes one or more real-world objects and one or more computer-generated (CG) objects. The scene can be characterized by scene geometry at a first time. Such an example generates shadow information for each of one or more shadows cast by the CG objects of the scene onto one or more real-world objects in the scene, based on the scene geometry at the first time, wherein the shadow information includes: i) shadow factors, each shadow factor describing the light attenuation effect of the shadow cast on the surface of the real-world object and incorporated into a shading atlas of the scene, and ii) and a set of identifiers, each identifier in the set identifying the real-world object surface of the scene affected by the shadow factors. Such an example encodes the shading atlas into a video data stream. Such an example sends the set of identifiers and the video stream encoded with the shading atlas to a second device over a network. Example 10 includes Example 9, wherein the video stream is a Moving Picture Experts Group (MPEG) transport stream. Example 11 includes any of Examples 9-10. Such examples include receiving second device scene geometry from the second device via the network prior to the generation. In such examples, the scene geometry at the first time is at least partially based on the second device scene geometry. Example 12 includes any of Examples 9-11, wherein the second device scene geometry includes one or more of the following: pose information of the second device, 2D information captured by the second device, or mesh information of the second device. Example 13 includes any of Examples 9-12, wherein the second device is one of a video perspective device or an optical perspective device.

[0100] In Example 14, the techniques disclosed herein may be methods for graphics processing, one or more apparatuses, or computer-readable media storing computer-executable code. In such an example, the scene includes one or more real-world objects and one or more computer-generated (CG) objects. The scene can be characterized by scene geometry at a first time. Such an example includes receiving, by a second device, from a first device and via a network, i) a set of identifiers and ii) a video stream encoded with a shading atlas. In such an example, the shading atlas includes a plurality of shadow factors, each shadow factor describing the light attenuation effect of shadows cast by CG objects of the scene onto one or more real-world objects of the scene, and each identifier in the set identifies a real-world object of the scene affected by the shadow factors. In such an example, the second device may perform the following operations: decode the video stream; and render and display frames of the scene according to: i) the shadow information, and ii) the scene geometry at a time later than the first time of rendering. Example 15 includes Example 14, wherein the video stream is a Moving Picture Experts Group (MPEG) transport stream. Example 16 includes any of Examples 14-15. In such an example, prior to the reception, the second device may transmit its scene geometry to the first device and via the network. In such an example, the scene geometry at the first time is at least partially based on the transmitted second device scene geometry. Example 17 includes any of Examples 14-16, wherein the transmitted second device scene geometry includes one or more of the following: pose information of the second device, 2D information captured by the second device, or mesh information of the second device. Example 18 includes any of Examples 14-17, wherein the second device is a video perspective device or an optical perspective device.

[0101] The themes described herein can be implemented to achieve one or more benefits or advantages. For example, the described graphics processing techniques can be used by a server, client, GPU, CPU, or some other processor capable of performing graphics processing to implement the split rendering techniques described herein. This can also be implemented at a lower cost compared to other graphics processing techniques. Furthermore, the graphics processing techniques in this paper can improve or accelerate data processing or execution. Further, the graphics processing techniques in this paper can improve resource or data utilization and / or resource efficiency. Moreover, examples of the techniques disclosed herein can utilize split rendering processes that can improve the accuracy of handling shadows in split rendering with both real-world and augmented content.

[0102] According to this disclosure, unless otherwise specified in the context, the term "or" can be interpreted as "and / or". Additionally, while phrases such as "one or more" or "at least one" may be used for some features disclosed herein but not for others, the absence of such language in a feature can be interpreted as implying such a meaning unless otherwise specified in the context.

[0103] In one or more examples, the functionality described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term “processing unit” has been used throughout this disclosure, such a processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any functionality, processing unit, technique, or other module described herein is implemented in software, the functionality, processing unit, technique, or other module described herein may be stored on or transmitted via a computer-readable medium as one or more instructions or code. A computer-readable medium may include a computer data storage medium or a communication medium, the communication medium including any medium that facilitates the transfer of a computer program from one place to another. In this way, a computer-readable medium may generally correspond to: (1) a tangible computer-readable storage medium that is non-transitory; or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures used to implement the techniques described herein. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically copy data magnetically, while optical discs use lasers to copy data optically. Combinations of the foregoing should also be included within the scope of computer-readable media. Computer program products may include computer-readable media.

[0104] The code can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), arithmetic logic units (ALUs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, the techniques can be sufficiently implemented in one or more circuit or logic elements.

[0105] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but they do not necessarily need to be implemented through different hardware units. Specifically, as described above, the various units can be combined in any hardware unit, or provided by a batch of interoperable hardware units (including one or more processors as described above) combined with suitable software and / or firmware.

[0106] Various examples have been described. These and other examples are within the scope of the appended claims.

Claims

1. A computer graphics processing method, comprising: For a scene comprising one or more real-world objects and one or more computer-generated CG objects, the scene is characterized by a first scene geometry in a first-time context: A first device generates shadow information for each shadow in one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene, based on the first scene geometry at the first time, wherein the shadow information includes: Shadow factors, each describing the light attenuation effect of shadows cast on the surface of a real-world object, are incorporated into the shading atlas of the scene. A set of identifiers, each identifier in the set identifying a real-world object surface in the scene affected by a shadow factor; and The shadow information is received by the second device from the first device via a network; The second device renders and displays frames of the scene based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

2. The method according to claim 1, wherein, The first device is a server.

3. The method according to claim 1, wherein, The method further includes: Prior to the rendering, the first device encodes the shading atlas into the video stream; The first device sends the set of identifiers and the video stream encoded with the shading atlas to the second device via the network; The second device receives the set of identifiers and the video stream encoded with the shading atlas from the first device and via the network; and The video stream is decoded by the second device.

4. The method according to claim 3, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

5. The method according to claim 3: The method further includes: Before generating shadow information: The geometry of the scene for the second device is determined by the second device; The second device sends the scene geometry of the second device to the first device through the network; and The first device receives the second device scene geometry sent by the second device through the network; The first scene geometry at the first time is at least partially based on the second device scene geometry.

6. The method according to claim 5, wherein, The scene geometry of the second device includes one or more of the following: the pose information of the second device, the image information captured by the second device, or the mesh information of the second device.

7. The method according to claim 1, wherein, The second device is either a video perspective device or an optical perspective device.

8. The method according to claim 1, wherein, The shadow information for generating each of one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene includes: at least partially blocking one or more real light sources in the scene.

9. A computer graphics processing method, comprising: For a scene comprising one or more real-world objects and one or more computer-generated CG objects, the scene is characterized by a first scene geometry in a first-time context: Based on the first scene geometry at the first time, shadow information is generated for each shadow in one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene, wherein the shadow information includes: i) shadow factors, each shadow factor describing the light attenuation effect of the shadow cast on the surface of the real-world object and being incorporated into the shading atlas of the scene, and ii) a set of identifiers, each identifier in the set identifying the real-world object surface of the scene affected by the shadow factors; Encode the shading atlas into the video stream; and The identifier set and the video stream encoded with the color atlas are sent to a second device via a network. The frames of the scene are rendered and displayed based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

10. The method according to claim 9, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

11. The method according to claim 9: The method further includes: Prior to generating the shadow information, the scene geometry of the second device is received from the second device via the network; The first scene geometry at the first time is at least partially based on the second device scene geometry.

12. The method according to claim 11, wherein, The scene geometry of the second device includes one or more of the following: the pose information of the second device, the two-dimensional information captured by the second device, or the mesh information of the second device.

13. The method according to claim 9, wherein, The second device is either a video perspective device or an optical perspective device, and the second device renders and displays frames of the scene based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

14. A computer graphics method, comprising: For a scene comprising one or more real-world objects and one or more computer-generated CG objects, the scene is characterized by a first scene geometry in a first-time context: The second device receives shadow information from the first device and via a network, the shadow information including: i) a set of identifiers and ii) a video stream encoded with a shading atlas, wherein: The shading atlas includes multiple shadow factors, each shadow factor describing the light attenuation effect of shadows cast by CG objects in the scene onto one or more real-world objects in the scene, and Each identifier in the set identifies a real-world object in the scene that is affected by the shadow factor; The video stream is decoded by the second device; and The second device renders and displays frames of the scene based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

15. The method according to claim 14, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

16. The method according to claim 14: The method further includes: Before receiving the shadow information, the second device sends the scene geometry of the second device to the first device and through the network. The first scene geometry at the first time is at least partially based on the second device scene geometry transmitted.

17. The method according to claim 16, wherein, The transmitted second device scene geometry includes one or more of the following: the pose information of the second device, the two-dimensional information captured by the second device, or the mesh information of the second device.

18. The method according to claim 14, wherein, The second device is either a video perspective device or an optical perspective device.

19. An apparatus for graphics processing, comprising: Memory; as well as At least one processor, coupled to the memory and configured to: handle a scene comprising one or more real-world objects and one or more computer-generated CG objects, characterized by a first-time scene geometry: Based on the first scene geometry at the first time, shadow information is generated for each shadow in one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene, wherein the shadow information includes: i) shadow factors, each shadow factor describing the light attenuation effect of the shadow cast on the surface of the real-world object and being incorporated into the shading atlas of the scene, and ii) a set of identifiers, each identifier in the set identifying the real-world object surface of the scene affected by the shadow factors; Encode the shading atlas into the video stream; and The identifier set and the video stream encoded with the color atlas are sent to a second device via a network. The frames of the scene are rendered and displayed based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

20. The apparatus according to claim 19, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

21. The apparatus according to claim 19, wherein, The at least one processor is further configured to receive second device scene geometry from the second device via the network before generating the shadow information, wherein the first scene geometry at the first time is at least partially based on the second device scene geometry.

22. The apparatus according to claim 21, wherein, The scene geometry of the second device includes one or more of the following: the pose information of the second device, the two-dimensional information captured by the second device, or the mesh information of the second device.

23. The apparatus according to claim 19, wherein, The second device is either a video perspective device or an optical perspective device, and the second device renders and displays frames of the scene based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

24. A computer graphics device, comprising: Memory; as well as At least one processor, coupled to the memory and configured to: handle a scene comprising one or more real-world objects and one or more computer-generated CG objects, characterized by a first-time scene geometry: Shadow information is received from a first device and via a network, the shadow information including: i) a set of identifiers, and ii) a video stream encoded with a shading atlas, wherein: The shading atlas includes multiple shadow factors, each shadow factor describing the light attenuation effect of shadows cast by CG objects in the scene onto one or more real-world objects in the scene, and Each identifier in the set identifies a real-world object in the scene that is affected by the shadow factor; Decode the video stream; and The scene frames are rendered and displayed based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

25. The apparatus according to claim 24, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

26. The apparatus according to claim 24, wherein, The at least one processor is further configured to: transmit current scene geometry to the first device and via the network before receiving the shadow information, wherein the first scene geometry at the first time is at least partially based on the transmitted current scene geometry.

27. The apparatus according to claim 26, wherein, The transmitted current scene geometry includes one or more of the following: the pose information of the rendering processor's device, the 2D information captured by the rendering processor's device, or the mesh information of the rendering processor's device.

28. The apparatus according to claim 24, wherein, The device of the rendering processor is either a video perspective device or an optical perspective device.

29. An apparatus for computer graphics processing, comprising: For a scene comprising one or more real-world objects and one or more computer-generated CG objects, the scene is characterized by a first scene geometry in a first-time context: A unit for generating shadow information for each shadow in one or more shadows cast by CG objects of the scene onto one or more real-world objects of the scene, based on the first scene geometry at the first time, wherein the shadow information includes: i) shadow factors, each shadow factor describing the light attenuation effect of the shadow cast on the surface of the real-world object and being incorporated into the shading atlas of the scene, and ii) a set of identifiers, each identifier in the set identifying the real-world object surface of the scene affected by the shadow factors; Units for encoding the shading atlas into a video stream; and A unit for sending the identifier set and the video stream encoded with the color atlas to a second device via a network. The frames of the scene are rendered and displayed based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

30. The apparatus according to claim 29, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

31. The apparatus according to claim 29, wherein, The apparatus further includes: a unit for receiving second device scene geometry from the second device via the network before generating the shadow information; and wherein the first scene geometry at the first time is at least partially based on the second device scene geometry.

32. The apparatus according to claim 31, wherein, The scene geometry of the second device includes one or more of the following: the pose information of the second device, the two-dimensional information captured by the second device, or the mesh information of the second device.

33. The apparatus according to claim 29, wherein, The second device is either a video perspective device or an optical perspective device, and the second device renders and displays frames of the scene based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

34. An apparatus for computer graphics processing, comprising: For a scene comprising one or more real-world objects and one or more computer-generated CG objects, the scene is characterized by a first scene geometry in a first-time context: A unit for receiving shadow information from a first device by a second device and via a network, the shadow information comprising: i) a set of identifiers, and ii) a video stream encoded with a shading atlas, wherein: The shading atlas includes multiple shadow factors, each shadow factor describing the light attenuation effect of shadows cast by CG objects in the scene onto one or more real-world objects in the scene, and Each identifier in the set identifies a real-world object in the scene that is affected by the shadow factor; A unit for decoding the video stream by the second device; and A unit for rendering and displaying frames of the scene by the second device according to: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

35. The apparatus according to claim 34, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

36. The apparatus according to claim 34, wherein, The apparatus further includes: a unit for transmitting second device scene geometry from the second device to the first device and via the network prior to receiving shadow information; and wherein the first scene geometry at the first time is at least partially based on the transmitted second device scene geometry.

37. The apparatus according to claim 36, wherein, The transmitted second device scene geometry includes one or more of the following: the pose information of the second device, the two-dimensional information captured by the second device, or the mesh information of the second device.

38. The apparatus according to claim 34, wherein, The second device is either a video perspective device or an optical perspective device.

39. A computer-readable medium storing computer-executable code for graphics processing, comprising code for performing the following operations: For a scene comprising one or more real-world objects and one or more computer-generated CG objects, the scene is characterized by a first scene geometry in a first-time context: The first device generates shadow information for each shadow in one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene, based on the first scene geometry at the first time, wherein... The shadow information includes: Shadow factors, each describing the light attenuation effect of shadows cast on the surface of a real-world object, are incorporated into the shading atlas of the scene. A set of identifiers, each identifier in the set identifying a real-world object surface in the scene affected by a shadow factor; and The shadow information is received by the second device from the first device via a network; and The second device renders and displays frames of the scene based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

40. The computer-readable medium according to claim 39, wherein, The first device is a server.

41. The computer-readable medium according to claim 39, wherein, The computer-readable medium also stores computer-executable code for performing the following operations: Prior to the rendering, the first device encodes the shading atlas into the video stream; The first device sends the set of identifiers and the video stream encoded with the shading atlas to the second device via the network; The second device receives from the first device and via the network a list of identifiers and a video stream encoded with the color atlas; as well as The video stream is decoded by the second device.

42. The computer-readable medium according to claim 41, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

43. The computer-readable medium according to claim 41: The computer-readable medium also stores computer-executable code for performing the following operations before generating the shadow information: The geometry of the scene for the second device is determined by the second device; The second device sends the scene geometry of the second device to the first device through the network; and The first device receives the second device scene geometry sent by the second device through the network; in, The first scene geometry at the first time is at least partially based on the second device scene geometry.

44. The computer-readable medium according to claim 43, wherein, The scene geometry of the second device includes one or more of the following: the pose information of the second device, the image information captured by the second device, or the mesh information of the second device.

45. The computer-readable medium according to claim 39, wherein, The second device is either a video perspective device or an optical perspective device.

46. ​​A computer-readable medium storing computer-executable code for graphics processing, comprising code for performing the following operations: For a scene comprising one or more real-world objects and one or more computer-generated CG objects, the scene is characterized by a first scene geometry in a first-time context: Based on the first scene geometry at the first time, shadow information is generated for each shadow in one or more shadows cast by CG objects of the scene onto one or more real-world objects in the scene, wherein, The shadow information includes: i) shadow factors, each shadow factor describing the light attenuation effect of shadows cast on the surface of a real-world object and being incorporated into the shading atlas of the scene; and ii) a set of identifiers, each identifier in the set identifying the real-world object surface of the scene affected by the shadow factors. Encode the shading atlas into the video stream; and The identifier set and the video stream encoded with the color atlas are sent to a second device via a network. The frames of the scene are rendered and displayed based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

47. The computer-readable medium of claim 46, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

48. The computer-readable medium according to claim 46: The computer-readable medium also stores computer-executable code for performing the following operations: receiving second device scene geometry from the second device via the network before generating shadow information; in, The first scene geometry at the first time is at least partially based on the second device scene geometry.

49. The computer-readable medium according to claim 48, wherein, The scene geometry of the second device includes one or more of the following: the pose information of the second device, the two-dimensional information captured by the second device, or the mesh information of the second device.

50. The computer-readable medium according to claim 46, wherein, The second device is either a video perspective device or an optical perspective device, and the second device renders and displays frames of the scene based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

51. A computer-readable medium storing computer-executable code for graphics processing, comprising code for performing the following operations: For a scene comprising one or more real-world objects and one or more computer-generated CG objects, the scene is characterized by a first scene geometry in a first-time context: The shadow information is received by the second device from the first device and via the network, the shadow information including: i) a set of identifiers, and ii) a video stream encoding the shading atlas, wherein: The shading atlas includes multiple shadow factors, each shadow factor describing the light attenuation effect of shadows cast by CG objects in the scene onto one or more real-world objects in the scene, and Each identifier in the set identifies a real-world object in the scene that is affected by the shadow factor; The video stream is decoded by the second device; and The second device renders and displays frames of the scene based on: i) the shadow information, and ii) the second scene geometry at a rendering time later than the first time.

52. The computer-readable medium according to claim 51, wherein, The video stream is a Moving Picture Experts Group (MPEG) transport stream.

53. The computer-readable medium according to claim 51: The computer-readable medium also stores computer-executable code for performing the following operations: prior to receiving the shadow information, the second device sends the second device scene geometry to the first device and through the network; in, The first scene geometry at the first time is at least partially based on the second device scene geometry transmitted.

54. The computer-readable medium according to claim 53, wherein, The transmitted second device scene geometry includes one or more of the following: the pose information of the second device, the two-dimensional information captured by the second device, or the mesh information of the second device.

55. The computer-readable medium according to claim 51, wherein, The second device is either a video perspective device or an optical perspective device.

Images