Image space function transfer

Abstract

The present disclosure relates to methods and apparatuses for graphics processing at a server and / or a client device. In some aspects, the apparatus can transform application data for at least one frame, the application data corresponding to one or more image functions or one or more data channels. The apparatus can also encode the application data for at least one frame, the application data associated with a data stream, the application data encoded via a video encoding process. The apparatus can also transmit the encoded application data for at least one frame. Further, the apparatus can receive the application data for at least one frame, the application data associated with a data stream. The apparatus can also decode the application data for at least one frame, and transform the application data for at least one frame.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 067,123, entitled “METHODS AND APPARATUS FOR IMAGE-SPACE FUNCTION TRANSMISSION”, filed August 18, 2020, and U.S. Patent Application No. 17 / 400,048, entitled “IMAGE-SPACE FUNCTION TRANSMISSION”, filed August 11, 2021, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] This disclosure relates generally to processing systems, and more specifically to one or more techniques for graphics processing. Background Technology

[0004] Computing devices frequently utilize graphics processing units (GPUs) to accelerate the rendering of graphics data used for display. These computing devices can include, for example, computer workstations, mobile phones such as so-called smartphones, embedded systems, personal computers, tablet computers, and video game consoles. A GPU runs a graphics processing pipeline comprising one or more processing stages that work together to execute graphics processing commands and output frames. A central processing unit (CPU) controls the operation of the GPU by issuing one or more graphics processing commands to it. Modern CPUs are typically capable of running multiple applications simultaneously, each of which may require the GPU during its execution. Devices that provide content for visual presentation on a display typically include a GPU.

[0005] Typically, a device's GPU is configured to perform processing within the graphics processing pipeline. However, with the advent of wireless communication and smaller handheld devices, the need for improved graphics processing continues to grow. Summary of the Invention

[0006] The following is a brief overview of one or more aspects to provide a basic understanding of them. This overview is not an exhaustive summary of all anticipated aspects, nor is it intended to identify key elements of all aspects or describe the scope of any or all aspects. The sole purpose of this overview is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that follows.

[0007] In aspects of the disclosure, a method, a computer-readable medium, and an apparatus are provided. The apparatus can be a server, a client device, a central processing unit (CPU), a graphics processing unit (GPU), or any apparatus that can perform graphics processing. The apparatus can generate application data for at least one frame. The apparatus can also transform the application data for the at least one frame, the transformed application data corresponding to one or more image functions or one or more data channels. The apparatus can also encode the application data for the at least one frame, the application data being associated with a data stream, the application data being encoded via a video encoding process. In addition, the apparatus can assign the encoded application data to one or more video frames or one or more MPEG frames. The apparatus can also store the encoded application data in the one or more video frames or the one or more MPEG frames. Furthermore, the apparatus can divide the one or more video frames into at least one sub-region associated with color data or at least one other sub-region associated with application data. The apparatus can also transmit the encoded application data for the at least one frame.

[0008] In another aspect of the disclosure, a method, a computer-readable medium, and an apparatus are provided. The apparatus can be a server, a client device, a central processing unit (CPU), a graphics processing unit (GPU), or any apparatus that can perform graphics processing. The apparatus can receive application data for at least one frame, the application data being associated with a data stream. The apparatus can also decode the application data for the at least one frame, the application data being decoded via a video decoding process. The apparatus can also transform the application data for the at least one frame, the transformed application data corresponding to one or more image functions or one or more data channels. In addition, the apparatus can display the application data for the at least one frame at a display panel.

[0009] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a block diagram illustrating an example content generation system in accordance with one or more techniques of this disclosure.

[0011] Figure 2 illustrates an example of a color atlas in accordance with one or more techniques of this disclosure.

[0012] Figure 3 illustrates a diagram of an example data encoding process in accordance with one or more techniques of this disclosure.

[0013] Figure 4 illustrates a diagram of an example data decoding process in accordance with one or more techniques of this disclosure.

[0014] Figure 5 A diagram illustrating an example data allocation process is shown.

[0015] Figure 6 A diagram illustrating an example data allocation process is shown.

[0016] Figure 7 A diagram illustrating an example data allocation process is shown.

[0017] Figure 8 A communication flow diagram illustrating example communications between components in accordance with one or more techniques of the present disclosure is shown.

[0018] Figure 9 An example flow diagram illustrating an example method in accordance with one or more techniques of the present disclosure is shown.

[0019] Figure 10 An example flow diagram illustrating an example method in accordance with one or more techniques of the present disclosure is shown.

[0020] Figure 11 An example flow diagram illustrating an example method in accordance with one or more techniques of the present disclosure is shown.

[0021] Figure 12 An example flow diagram illustrating an example method in accordance with one or more techniques of the present disclosure is shown. DETAILED DESCRIPTION

[0022] In some aspects of split rendering, rendering can be performed using red (R), green (G), blue (B) (RGB) color data. A hardware video encoder can convert the RGB color data to luminance (Y) chroma (UV) (YUV) color data. To use the hardware, auxiliary data (AuxD) can be provided as RGB color data and communicated as YUV color data. In some instances, certain YUV color data (e.g., YUV 4:2:0) can contain a portion of the data, e.g., half of the data, as compared to RGB color data (e.g., RGB 4:4:4). The Y portion of the data can be sent at full resolution, and the U and V portions can be downsampled by a factor of 2, which can result in ¼ of the resolution. Aspects of the present disclosure can encode the AuxD so that it can be recovered to a known precision after a lossy transform. Aspects of the present disclosure can also convert and encode the RGB color data to YUV color data, and then send the YUV color data in split rendering. Further, aspects of the present disclosure can receive YUV color data in split rendering, and then decode and convert the YUV color data back to RGB color data.

[0023] Various aspects of the systems, apparatuses, computer program products, and methods are more fully described below with reference to the figures. The present disclosure may, however, be embodied in many different forms and should not be construed as limited to the any specific structure or function presented throughout this disclosure. Rather, these aspects are provided as illustrative so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Based on the teachings herein one skilled in the art should appreciate that the scope of the disclosure is intended to cover any aspect of the systems, apparatuses, computer program products, and methods disclosed herein, whether implemented independently of, or combined with, other aspects of the disclosure. For example, an apparatus can be implemented or a method can be practiced using any number of the aspects set forth herein. In addition, the scope of the disclosure is intended to cover such an apparatus or method which is practiced using, in addition to or in place of the aspects set forth herein, other structures, functionalities, or structures and functions practiced by the aspects now disclosed. Any aspect disclosed herein can be implemented by one or more elements of a claim.

[0024] Although various aspects are described herein, many variations and permutations of these aspects fall within the scope of the disclosure. While some potential benefits and advantages of aspects of the disclosure are mentioned, the scope of the disclosure is not intended to be limited to particular benefits, uses, or objectives. Rather, aspects of the disclosure are intended to be broadly applicable to different wireless technologies, system configurations, networks, and transmission protocols, some of which are illustrated by way of example in the accompanying drawings and description below. The detailed description and drawings are merely illustrative of the present disclosure rather than limiting, the scope of the present disclosure being defined by the appended claims and their equivalents.

[0025] Several aspects are presented with reference to various apparatuses and methods. These apparatuses and methods will be described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented as hardware or software depends on the particular application and design constraints imposed on the overall system.

[0026] As an example, an element, or any portion of an element, or any combination of elements can be implemented as a "processing system" that includes one or more processors. Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on a chip (SoC), baseband processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. One or more processors in the processing system can execute software. Software can be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, runtimes, processes, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. The term application can refer to software. As described herein, one or more techniques can refer to an application, i.e., software, configured to perform one or more functions. In these examples, an application can be stored on a memory, e.g., on-chip memory of a processor, system memory, or any other memory. Hardware described herein, such as a processor, can be configured to execute the application. For example, an application can be described as including code that, when executed by hardware, causes the hardware to perform one or more techniques described herein. As an example, the hardware can access the code from a memory and execute the code accessed from the memory to perform one or more techniques described herein. In some examples, components are identified in this disclosure. In these examples, a component can be hardware, software, or a combination thereof. A component can be a separate component or a subcomponent of a single component.

[0027] Accordingly, in one or more examples described herein, the described functions can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM), diskette, a hard disk drive, magnetic tape, other magnetic media, a flash memory or other solid-state memory, a combination of any of the foregoing, or any other medium that can be used to store computer-executable or other software code in the form of instructions or data structures and that can be accessed by a computer.

[0028] In general, this disclosure describes techniques for having a graphics processing pipeline in a single device or multiple devices, improving rendering of graphics content, and / or reducing the load on a processing unit (i.e., any processing unit configured to perform one or more techniques described herein, such as a GPU) that performs graphics processing. For example, this disclosure describes techniques for graphics processing in any device that utilizes graphics processing. Other example benefits are described throughout this disclosure.

[0029] As used herein, an example of the term “content” can refer to “graphics content,” “a product of a 3D graphics design,” a rendition thereof, i.e., an “image,” and vice versa. This is true regardless of whether the term is used as an adjective, a noun, or other part of speech. In some examples, as used herein, the term “graphics content” can refer to content produced by one or more processes of a graphics processing pipeline. In some examples, as used herein, the term “graphics content” can refer to content produced by a processing unit configured to perform graphics processing. In some examples, as used herein, the term “graphics content” can refer to content produced by a graphics processing unit.

[0030] In some examples, as used herein, the term “display content” can refer to content generated by a processing unit configured to perform display processing. In some examples, as used herein, the term “display content” can refer to content generated by a display processing unit. Graphics content can be processed into display content. For example, a graphics processing unit can output graphics content, such as a frame, to a buffer (which can be referred to as a frame buffer). A display processing unit can read the graphics content (such as one or more frames) from the buffer and perform one or more display processing techniques on it to generate display content. For example, a display processing unit can be configured to perform compositing on one or more rendered layers to generate a frame. As another example, a display processing unit can be configured to composite, blend, or otherwise combine two or more layers into a single frame. A display processing unit can be configured to perform scaling, e.g., zooming in or out, on a frame. In some examples, a frame can refer to a layer. In other examples, a frame can refer to two or more layers that have been blended together to form a frame, i.e., a frame includes two or more layers, and the frame including the two or more layers can subsequently be blended.

[0031] Figure 1is a block diagram illustrating an example system 100 configured to implement one or more techniques of the present disclosure. The system 100 includes a device 104. The device 104 can include one or more components or circuits to perform the various functions described herein. In some examples, one or more components of the device 104 can be components of a SOC. The device 104 can include one or more components configured to perform one or more techniques of the present disclosure. In the illustrated example, the device 104 can include a processing unit 120, a content encoder / decoder 122, and a system memory 124. In some aspects, the device 104 can include a number of optional components, e.g., a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131. Reference to the displays 131 can refer to one or more displays 131. For example, the displays 131 can include a single display or multiple displays. The displays 131 can include a first display and a second display. The first display can be a left eye display and the second display can be a right eye display. In some examples, the first and second displays can receive different frames for presentation thereon. In other examples, the first and second displays can receive the same frames for presentation thereon. In further examples, the results of the graphics processing can not be displayed on the device, e.g., the first and second displays can not receive any frames for presentation thereon. Instead, the frames or graphics processing results can be communicated to another device. In some aspects, this can be referred to as split rendering.

[0032] The processing unit 120 can include an internal memory 121. The processing unit 120 can be configured to perform graphics processing, such as in a graphics processing pipeline 107. The content encoder / decoder 122 can include an internal memory 123. In some examples, the device 104 can include a display processor, such as the display processor 127, to perform one or more display processing techniques on one or more frames generated by the processing unit 120 prior to presentation by the one or more displays 131. The display processor 127 can be configured to perform display processing. For example, the display processor 127 can be configured to perform one or more display processing techniques on one or more frames generated by the processing unit 120. The one or more displays 131 can be configured to display or otherwise present the frames processed by the display processor 127. In some examples, the one or more displays 131 can include one or more of a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head-mounted display, or any other type of display device.

[0033] Memory external to processing unit 120 and content encoder / decoder 122, such as system memory 124, can be accessed by processing unit 120 and content encoder / decoder 122. For example, processing unit 120 and content encoder / decoder 122 can be configured to read from and / or write to external memory, such as system memory 124. Processing unit 120 and content encoder / decoder 122 can be communicatively coupled to system memory 124 by a bus. In some examples, processing unit 120 and content encoder / decoder 122 can be communicatively coupled to each other by a bus or a different connection.

[0034] Content encoder / decoder 122 can be configured to receive graphical content from any source, such as system memory 124 and / or communication interface 126. System memory 124 can be configured to store received encoded or decoded graphical content. Content encoder / decoder 122 can be configured to receive encoded or decoded graphical content, for example, from system memory 124 and / or communication interface 126 in the form of encoded pixel data. Content encoder / decoder 122 can be configured to encode or decode any graphical content.

[0035] Internal memory 121 or system memory 124 can include one or more volatile or non-volatile memories or storage devices. In some examples, internal memory 121 or system memory 124 can include RAM, SRAM, DRAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, a magnetic data medium, or an optical storage medium, or any other type of memory.

[0036] According to some examples, internal memory 121 or system memory 124 can be a non-transitory storage medium. The term “non-transitory” can indicate that the storage medium is not implemented using a carrier wave or a propagating signal. However, the term “non-transitory” should not be interpreted to mean that internal memory 121 or system memory 124 is not movable or that its contents are static. As one example, system memory 124 can be removed from device 104 and moved to another device. As another example, system memory 124 can not be removable from device 104.

[0037] The processing unit 120 can be a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), or any other processing unit that can be configured to perform graphics processing. In some examples, the processing unit 120 can be integrated into a motherboard of the device 104. In some examples, the processing unit 120 can be present on a graphics card that is installed in a port of the motherboard of the device 104, or can be otherwise incorporated within a peripheral device that is configured to interoperate with the device 104. The processing unit 120 can include one or more processors, such as one or more microprocessors, GPUs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuitry, or any combination thereof. If these technologies are partially implemented in software, the processing unit 120 can store instructions for the software in suitable, non-transitory computer-readable storage media (e.g., an internal memory 121) and can execute the instructions in hardware using one or more processors to perform the techniques of the present disclosure. Any of the above-described (including hardware, software, a combination of hardware and software, etc.) can be considered one or more processors.

[0038] The content encoder / decoder 122 can be any processing unit configured to perform content decoding. In some examples, the content encoder / decoder 122 can be integrated into a motherboard of the device 104. The content encoder / decoder 122 can include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuitry, or any combination thereof. If these technologies are partially implemented in software, the content encoder / decoder 122 can store instructions for the software in suitable, non-transitory computer-readable storage media (e.g., an internal memory 123) and can execute the instructions in hardware using one or more processors to perform the techniques of the present disclosure. Any of the above-described (including hardware, software, a combination of hardware and software, etc.) can be considered one or more processors.

[0039] In some aspects, system 100 can include optional communication interface 126. Communication interface 126 can include receiver 128 and transmitter 130. Receiver 128 can be configured to perform any of the receiving functions described herein with respect to device 104. Further, receiver 128 can be configured to receive information from another device, e.g., eye or head position information, rendering commands, or location information. Transmitter 130 can be configured to perform any of the transmitting functions described herein with respect to device 104. For example, transmitter 130 can be configured to transmit information to another device, which can include a request for content. Receiver 128 and transmitter 130 can be combined into transceiver 132. In such examples, transceiver 132 can be configured to perform any of the receiving functions and / or transmitting functions described herein with respect to device 104.

[0040] Referring again to Figure 1 In certain aspects, processing unit 120 can include determination component 198 configured to generate application data for at least one frame. Determination component 198 can be further configured to transform the application data for the at least one frame, the transformed application data corresponding to one or more image functions or one or more data channels. Determination component 198 can be further configured to encode the application data for the at least one frame, the application data associated with a data stream, the application data encoded via a video encoding process. Determination component 198 can be further configured to assign the encoded application data to one or more video frames or one or more MPEG frames. Determination component 198 can be further configured to store the encoded application data in the one or more video frames or the one or more MPEG frames. Determination component 198 can be further configured to divide the one or more video frames into at least one sub-region associated with color data or at least one other sub-region associated with application data. Determination component 198 can be further configured to transmit the encoded application data for the at least one frame.

[0041] Referring again to Figure 1 In certain aspects, processing unit 120 can include determination component 198 configured to receive application data for at least one frame, the application data associated with a data stream. Determination component 198 can be further configured to decode the application data for the at least one frame, the application data decoded via a video decoding process. Determination component 198 can be further configured to transform the application data for the at least one frame, the transformed application data corresponding to one or more image functions or one or more data channels. Determination component 198 can be further configured to display the application data for the at least one frame at a display panel.

[0042] As described herein, a device, such as device 104, can refer to any device, apparatus, or system configured to perform one or more techniques described herein. For example, a device can be a server, a base station, a user device, a client device, a station, an access point, a computer (e.g., a personal computer, a desktop computer, a laptop computer, a tablet computer, a computer workstation, or a mainframe computer), an end product, an apparatus, a telephone, a smartphone, a server, a video game platform or console, a handheld device (e.g., a portable video game device or a personal digital assistant (PDA)), a wearable computing device (e.g., a smartwatch, an augmented reality device, or a virtual reality device), a non-wearable device, a display or display device, a television, a television set-top box, an intermediary network device, a digital media player, a video streaming device, a content streaming device, an in-vehicle computer, any mobile device, any device configured to generate graphical content, or any device configured to perform one or more techniques described herein. Processes herein can be described as being performed by a particular component (e.g., a GPU), but in additional embodiments, other components (e.g., a CPU) can be used to perform consistent with the disclosed embodiments.

[0043] A GPU can process multiple types of data or data packets in a GPU pipeline. For example, in some aspects, a GPU can process two types of data or data packets, such as context register packets and draw call data. A context register packet can be a set of global state information, such as information about global registers, shading programs, or constant data, which can dictate how a graphics context will be processed. For example, a context register packet can include information about a color format. In some aspects of a context register packet, there can be a bit that indicates which workload belongs to the context register. Further, there can be multiple functions or programs that run simultaneously and / or in parallel. For example, a function or program can describe a certain operation, such as a color mode or color format. Thus, a context register can define multiple states of a GPU.

[0044] A context state can be used to determine how a single processing unit, such as a vertex fetcher (VFD), a vertex shader (VS), a shader processor or geometry processor, and / or a processing unit runs in a mode. To do so, a GPU can use context registers and programming data. In some aspects, a GPU can generate a workload, such as a vertex or pixel workload, in a pipeline based on a context register definition of a mode or state. Certain processing units, such as a VFD, can use these states to determine certain functions, such as how vertices are combined. As these modes or states can change, a GPU can need to change the corresponding context. Further, a workload corresponding to a mode or state can follow a changing mode or state.

[0045] A GPU can render images in a number of different ways. In some cases, a GPU can render images using render or tiled rendering. In a tiled rendering GPU, an image can be divided or separated into different sections or tiles. After the image is divided, each section or tile can be rendered individually. A tiled rendering GPU can divide a computer graphics image into a grid format such that each section of the grid (i.e., a tile) can be rendered individually. In some aspects, during a binning pass, an image can be divided into different bins or tiles. Further, in the binning pass, different primitives can be shaded (e.g., using draw calls) in certain bins. In some aspects, during the binning pass, a visibility stream can be constructed in which visible primitives or draw calls can be identified.

[0046] In some aspects of rendering, there can be multiple processing stages or passes. For example, rendering can be performed in two passes, such as a visibility pass and a rendering pass. During the visibility pass, a GPU can input a rendering workload, record the location of primitives or triangles, and then determine which primitives or triangles fall into which portion of a frame. In some aspects of the visibility pass, the GPU can also identify or mark the visibility of each primitive or triangle in a visibility stream. During the rendering pass, the GPU can input the visibility stream and process a portion of the frame at a time. In some aspects, the visibility stream can be analyzed to determine which primitives are visible or not visible. As such, visible primitives can be processed. By doing so, the GPU can reduce unnecessary workload of processing or rendering primitives that are not visible.

[0047] In some aspects, rendering can be performed at multiple locations and / or on multiple devices, such as to divide a rendering workload between different devices. For example, rendering can be split between a server and a client device, which can be referred to as “split rendering.” In some instances, split rendering can be a method for bringing content to a user device or head-mounted display (HMD), where a portion of the graphics processing can be performed outside of the device or HMD, such as at a server.

[0048] Split rendering can be performed for a variety of different types of applications, such as virtual reality (VR) applications, augmented reality (AR) applications, and / or extended reality (XR) applications. In VR applications, the content displayed at a user device can correspond to artificial or animated content. In AR or XR content, a portion of the content displayed at a user device can correspond to real-world content, such as objects in the real world, and a portion of the content can be artificial or animated content. Furthermore, artificial or animated content and real-world content can be displayed in an optical see-through or video see-through device so that a user can view real-world objects as well as artificial or animated content at the same time. In some aspects, artificial or animated content can be referred to as augmentation content, and vice versa.

[0049] Split XR or AR systems can also introduce latency when transmitting rendered content to a client display. In some aspects, this latency can even be higher when rendering on a server compared to client rendering, but it can also enable more complex XR or AR applications. Furthermore, there can be a non-negligible latency between the time a client device’s pose is computed and the time content appears on a client display. For example, there can be an amount of latency in split XR or AR systems.

[0050] Figure 2 A shaded atlas 200 is shown in accordance with one or more techniques of the present disclosure. Some aspects of graphics processing can utilize a shaded atlas stream, such as via a Figure 2 In some examples, a shaded atlas stream can also be referred to as a vector stream. A shaded atlas stream can dynamically patch patches of primitives into a texture atlas. A shaded atlas stream can also patch a number of primitives or triangles (e.g., one (1) to three (3) primitives or triangles) assigned to a rectangle in the atlas.

[0051] In split rendering applications, a server can compute shading information in a shaded atlas. The shaded atlas can be encoded, such as Moving Picture Experts Group (MPEG) encoded, and streamed to a client device, such as a standalone head-mounted display (HMD). The client device can render a final view at a higher frame rate. Furthermore, shading can be sampled from the shaded atlas. As a user moves their head, novel views can be rendered. These novel views can use the latest atlas frame available at the client device.

[0052] Some aspects of split rendering can include auxiliary data (AuxD) in streaming rendering. AuxD can be provided as an image space function, such as transparency, depth, and / or high dynamic range exponent. AuxD can also be used for a portion of the original video stream. In some aspects, since MPEG data supported hardware can work with 3-channel video (e.g., YUV 4:2:0), AuxD can not use extended MPEG data, such as red (R), green (G), blue (B) (RGB), alpha (RGBA) data.

[0053] In some aspects of split rendering, rendering can be performed in RGB color data. Hardware video encoders can convert RGB color data to luminance (Y) chrominance (UV) (YUV) color data. To take advantage of certain types of hardware, AuxD can be provided as RGB color data. Further, AuxD can be transmitted as YUV color data. In some instances, certain YUV color data (e.g., YUV 4:2:0) can contain a portion of data, such as half of the data, as compared to RGB color data (e.g., RGB 4:4:4). For example, the Y portion of the data can be sent at full resolution, and the U and V portions can include resolutions that are downsampled by a factor (e.g., a factor of 2). This can result in a fraction of the resolution being transmitted, such as ¼ of the resolution.

[0054] Based on the above, it can be beneficial to encode auxiliary data (AuxD) so that it can be recovered to a known precision after a lossy transform. For example, it can be beneficial to convert and encode AuxD data to YUV color data, and then transmit the YUV color data in split rendering. It can also be beneficial to receive YUV color data in split rendering. Further, it can be beneficial to decode and convert the YUV color data back to AuxD data.

[0055] Aspects of the disclosure can encode AuxD so that it can be recovered to a known precision after a lossy transform. Aspects of the disclosure can also convert and encode AuxD data to YUV color data, and then transmit the YUV color data in split rendering. Further, aspects of the disclosure can receive YUV color data in split rendering. Aspects of the disclosure can also decode and convert the YUV color data back to AuxD color data.

[0056] Figure 3 FIG. 300 illustrates an example data encoding process in accordance with one or more techniques of the disclosure. Figure 3The diagram 300 in FIG. 3 includes original data 310, which includes RGB data 312 and auxiliary data 314. The diagram 300 also includes a compact YUV data layout 320 and a final YUV representation 340. As shown in FIG. 3, Figure 3 The RGB data 312 can be encoded at a server, as shown in FIG. 3. The auxiliary data 314 can then be interpreted as the compact YUV data layout 320. These steps can result in the final YUV representation 340, which can be ready for MPEG encoding. In some instances, Figure 3 The steps in FIG. 3 can be performed by a server.

[0057] As shown in FIG. 4, Figure 3 As shown in FIG. 3, aspects of the present disclosure can include examples of encoding a full-frame auxiliary channel. Aspects of the present disclosure can also interpret the auxiliary channel as a compact YUV 4:2:0 layout. Aspects of the present disclosure can also convert the YUV layout to RGB color data, and vice versa. Furthermore, aspects of the present disclosure can interleave the auxiliary RGB representation with regular RGB data, or interleave the auxiliary YUV representation with regular YUV data. Aspects of the present disclosure can also submit the YUV representation / data to MPEG encoding.

[0058] Figure 4 A diagram 400 is shown that illustrates an example data decoding process according to one or more techniques of the present disclosure. Figure 4 The diagram 400 in FIG. 4 includes a YUV representation 410, YUV color data 430 (i.e., a compact YUV layout or YUV segment), and an auxiliary channel 440. As shown in FIG. 4, Figure 4 As shown in FIG. 4, a client device can reverse the steps performed on a server in FIG. 3. The client device can decode the MPEG frame to YUV data, which can result in the YUV representation 410. Furthermore, the client device can convert the auxiliary YUV representation to YUV color data 430. The client device can then reassemble the auxiliary channel 440 from the YUV color data 430. By doing so, the auxiliary channel 440 can be recovered. Figure 3

[0059] Aspects of the present disclosure can also reduce or prevent loss of information during YUV color data conversion. As described herein, previous encoding steps can compensate for YUV subsampling. MPEG encoding can be lossy, for example, due to quantization in the discrete cosine transform (DCT) domain. In some aspects, for 8-bit input channels, AuxD can lose a certain number of bits (e.g., 1-2 bits) of precision. Furthermore, monotonicity of the function can be preserved. In some instances, the loss can be acceptable, for example, for alpha buffers. Furthermore, aspects of the present disclosure can adjust AuxD by a non-linear transform (e.g., a gamma curve) prior to transforming AuxD to obtain improved results. ​

[0060] Figure 5 FIG. 500 illustrates an example data allocation process according to one or more techniques of the disclosure. Figure 5 FIG. 500 includes YUV blocks 510 (e.g., YUV1, YUV2, YUV3, YUV4, and YUV5), AuxD blocks 520 (e.g., Aux1 and Aux2), Aux-YUV blocks 530 (e.g., Aux1-YUV and Aux2-YUV), and atlas 540. As shown in Figure 5 As shown in FIG. 500, aspects of the disclosure can include per-block auxiliary data in a texture atlas (e.g., atlas 540). For example, a texture atlas (e.g., atlas 540) can combine YUV blocks 510 (rectangular arrays of pixels) into an image container. The image container can be hardware MPEG-encodable. If AuxD is included in a block (e.g., AuxD blocks 520), aspects of the disclosure can mix AuxD blocks 520 and YUV blocks 510 in the same atlas (e.g., atlas 540). Additionally, YUV blocks 510 can be inserted directly into atlas 540. As shown in Figure 4 As shown in FIG. 500, AuxD blocks 520 can be converted to Aux-YUV blocks 530 (such as for a full AuxD frame) prior to insertion into atlas 540.

[0061] Figure 6 FIG. 600 illustrates an example data allocation process according to one or more techniques of the disclosure. Figure 6 FIG. 600 includes RGBA block 610, RGB block 620, and alpha block 630, each of which can be associated with a certain image (e.g., an image of a leaf). As shown in Figure 6 As shown in FIG. 600, aspects of the disclosure can include per-block allocation. In some instances, an additional image function can be allocated for each block in an atlas. For example, RGBA can be split into RGB + alpha, e.g., RGBA block 610 can be split into RGB block 620 and alpha block 630. In some aspects, alpha can be included in a separate AuxD block in the atlas. In some instances, if alpha is equal to one (1) for an entire RGBA block, an AuxD block can not be created.

[0062] Figure 7 FIG. 700 illustrates an example data allocation process according to one or more techniques of the disclosure. Figure 7 FIG. 700 includes alpha mask 710, alpha mask 720, Y block 722, U block 724, V block 726, alpha mask 730, and overlap allocation 732. Figure 7 An example of per-block allocation is shown. As shown in Figure 7 As shown in FIG. 700, in per-block allocation, aspects of the disclosure can include four (4) blocks with alpha masks. As shown inFigure 7 As shown in FIG. 7, the alpha mask 710 can be converted into different sized Y blocks 722 (e.g., original size), U blocks 724 (e.g., 4x original size), and V blocks 726 (e.g., 4x original size) to compensate for sub-sampling. Aspects of the present disclosure can use an overlapping allocation of Y, U, V blocks in the atlas, e.g., 2xY + lxU + lxV, in the same atlas block. For example, the overlapping allocation 732 can include two (2) Y blocks 722, one (1) U block 724, and one (1) V block 726.

[0063] Aspects of the present disclosure can also include parallel block allocation. In some aspects, the AuxD block allocation can extend existing block allocation in the atlas. The allocation can run in parallel on the GPU, e.g., in a compute shader. First, aspects of the present disclosure can determine how many AuxD blocks are used. The AuxD blocks can be placed in different channels and can result in partially filled atlas blocks. For example, partially filled blocks can be filled before new AuxD blocks are allocated. A certain number of blocks (e.g., 1 / 3 of the blocks) can be provided by allocating larger blocks (e.g., 4x larger blocks), and the number of blocks can use the U channel or the V channel. Further, a certain number of blocks (e.g., 2 / 3 of the blocks) can be provided by allocating blocks that are the same size as the input AuxD, and the number of blocks can use the Y channel. After determining how many blocks can be used, aspects of the present disclosure can use existing allocation strategies for vector streaming.

[0064] Aspects of the present disclosure can also include a number of use cases. For example, aspects of the present disclosure can include smooth transparency functions, e.g., windows, water, fog, fire, rain, decals, etc. Aspects of the present disclosure can include hard alpha masks, e.g., vegetation. Further, aspects of the present disclosure can include depth values, which can enable selective depth based warping. Further, aspects of the present disclosure can include a luminance multiplier, which can enable high dynamic range (HDR) streaming for regions with high dynamic range.

[0065] In some aspects of the present disclosure, any number of AuxD blocks can be combined in a block by block allocation strategy. For example, there can be transparency in some blocks, depth in other blocks, alpha masks in other blocks, and HDR in other blocks. Further, AuxD blocks can be combined in one atlas block within different channels.

[0066] Aspects of the present disclosure can include systems for transmitting additional image functions or data channels embedded directly into MPEG streams. Aspects of the present disclosure can also include systems that use sub-sampled channel (such as YUV 4:2:0) based encoding or decoding hardware to encode multiple full resolution functions in minimal space. Further, aspects of the present disclosure can include systems that transmit additional image functions for a subset of full frames. Aspects of the present disclosure can also include systems that perform block-wise allocation of video frames (similar to texture atlases) and store additional information compactly in encoded video frames. Aspects of the present disclosure can also include systems that perform efficient parallel block allocation of sub-sample blocks, e.g., using a size-based heuristic.

[0067] Figure 8 is a communication flow diagram 800 of graphics processing in accordance with one or more techniques of the present disclosure. As shown in Figure 8 The diagram 800 includes example communications between a server 802 and a client device 804 (e.g., a headset or HMD) in accordance with one or more techniques of the present disclosure.

[0068] At 810, the server 802 can generate application data for at least one frame. In some aspects, generating application data for at least one frame can include rendering application data for at least one frame, e.g., the server 802 can render application data for at least one frame. The application data can correspond to a set of values or a multi-variate function. The application data can be associated with a subset of at least one frame.

[0069] At 812, the server 802 can convert the application data for at least one frame, the application data corresponding to one or more image functions or one or more data channels. The one or more image functions can correspond to one or more two-dimensional (2D) image functions. Further, each of the one or more image functions can include a plurality of pixels, and each of the plurality of pixels can include one or more function values. The one or more function values can include at least one of one or more depth values, one or more textures, one or more values, texture coordinates, one or more transparency values, one or more screen masks, one or more normal vectors, one or more primitive identifiers (IDs), one or more global coordinate positions, or one or more displacement maps.

[0070] At 820, the server 802 can encode application data for the at least one frame, the application data being associated with a data stream, the application data being encoded via a video encoding process. In some aspects, encoding the application data for the at least one frame can include subsampling the application data for the at least one frame, e.g., the server 802 can subsample the application data for the at least one frame. The application data can be subsampled based on one or more subsampling channels, where at least one of the one or more subsampling channels can be a luminance (Y) chrominance (UV) (YUV) channel. Further, the data stream can be a Moving Picture Experts Group (MPEG) stream or a video stream. Further, the video encoding process can include Moving Picture Experts Group (MPEG) encoding.

[0071] At 830, the server 802 can distribute the encoded application data to one or more video frames or one or more MPEG frames. The encoded application data can be distributed block-wise for the one or more video frames or can be distributed in parallel blocks.

[0072] At 840, the server 802 can store the encoded application data in the one or more video frames or the one or more MPEG frames.

[0073] At 842, the server 802 can divide the one or more video frames into at least one subregion associated with color data or at least one other subregion associated with application data.

[0074] At 850, the server 802 can transmit the encoded application data for the at least one frame, e.g., application data 852. The application data can be red (R), green (G), blue (B) (RGB) color data before being converted, and the application data can be luminance (Y) chrominance (UV) (YUV) color data after being encoded.

[0075] At 860, the client device 804 can receive application data for the at least one frame, e.g., application data 852, where the application data can be associated with a data stream. The application data can correspond to a set of values or a multivariate function. Further, the application data can be associated with a subset of the at least one frame. The data stream can be a Moving Picture Experts Group (MPEG) stream.

[0076] At 870, the client device 804 can decode the application data of the at least one frame, the application data decoded via a video decoding process. In some aspects, decoding the application data of the at least one frame can include inverse subsampling the application data of the at least one frame, e.g., the client device 804 can inverse subsample the application data of the at least one frame. The application data can be inverse subsampled based on one or more subsampled channels, where at least one of the one or more subsampled channels can be a luminance (Y) chrominance (UV) (YUV) channel. The video decoding process can include a Motion Picture Experts Group (MPEG) decoding.

[0077] In some aspects, the decoded application data can be allocated for the one or more video frames. Further, the decoded application data can be stored in the one or more video frames. The decoded application data can be allocated block-wise for the one or more video frames or can be allocated in parallel blocks. The one or more video frames can be divided into at least one sub-region associated with color data or at least one other sub-region associated with application data.

[0078] At 880, the client device 804 can convert the application data of the at least one frame, the converted application data corresponding to one or more image functions or one or more data passes. The application data can be luminance (Y) chrominance (UV) (YUV) color data prior to being decoded and the application data can be red (R), green (G), blue (B) (RGB) color data after being converted. The one or more image functions can correspond to one or more two-dimensional (2D) image functions. Further, each of the one or more image functions can include a plurality of pixels and each of the plurality of pixels can include one or more function values. The one or more function values can include at least one of one or more depth values, one or more texture values, one or more texture coordinates, one or more transparency values, one or more screen masks, one or more normal vectors, one or more primitive identifiers (IDs), one or more global coordinate positions, or one or more displacement maps.

[0079] At 890, the client device 804 can display the application data of the at least one frame at a display panel.

[0080] Figure 9 An example flow diagram 900 illustrating an example method in accordance with one or more techniques of the present disclosure is shown. The method can be performed by an apparatus, such as a server, a client device, a CPU, a GPU, or an apparatus for graphics processing.

[0081] At 904, the apparatus can convert the application data of the at least one frame, the application data corresponding to one or more image functions or one or more data channels, as described in connection with 880.Figures 2 to 8 The server 802 can transform application data for at least one frame, the application data corresponding to one or more image functions or one or more data channels, as described in connection with the examples in Figure 8 The server 802 can transform application data for at least one frame, the application data corresponding to one or more image functions or one or more data channels, as described in connection with the examples in Figure 1 The processing unit 120 in the example in FIG. 12 can perform step 904. The one or more image functions can correspond to one or more two-dimensional (2D) image functions. Further, each of the one or more image functions can include a plurality of pixels, and each of the plurality of pixels can include one or more function values. The one or more function values can include at least one of one or more depth values, one or more textures, one or more values, texture coordinates, one or more transparency values, one or more screen masks, one or more normal vectors, one or more primitive identifiers (IDs), one or more global coordinate positions, or one or more displacement maps. The application data can correspond to a set of values or a multivariate function. The application data can be associated with a subset of at least one frame.

[0082] At 906, the apparatus can encode application data for at least one frame, the application data being associated with a data stream, the application data being encoded via a video encoding process, as described in connection with the examples in Figures 2 to 8 The server 802 can encode application data for at least one frame, the application data being associated with a data stream, the application data being encoded via a video encoding process, as described in connection with the examples in Figure 8 The server 802 can encode application data for at least one frame, the application data being associated with a data stream, the application data being encoded via a video encoding process, as described in connection with the examples in Figure 1 The processing unit 120 in the example in FIG. 12 can perform step 906. In some aspects, encoding the application data for at least one frame can include sub-sampling the application data for at least one frame, e.g., the apparatus can sub-sample the application data for at least one frame. The application data can be sub-sampled based on one or more sub-sampled channels, where at least one of the one or more sub-sampled channels can be a luminance (Y) chrominance (UV) (YUV) channel. Further, the data stream can be a Motion Picture Experts Group (MPEG) stream or a video stream. Further, the video encoding process can include Motion Picture Experts Group (MPEG) encoding.

[0083] At 914, the apparatus can transmit the encoded application data for at least one frame, as described in connection with the examples in Figures 2 to 8 The server 802 can transmit the encoded application data for at least one frame, as described in connection with the examples in Figure 8 The server 802 can transmit the encoded application data for at least one frame, as described in connection with the examples in Figure 1The processing unit 120 can execute step 914. The application data can be red (R), green (G), and blue (B) (RGB) color data before conversion, and the application data can be luminance (Y) and chrominance (UV) (YUV) color data after encoding.

[0084] Figure 10 An example flowchart 1000 is shown, illustrating an example method according to one or more techniques of this disclosure. This method can be executed by a device such as a server, client device, CPU, GPU, or a device for graphics processing.

[0085] At position 1002, the device can generate at least one frame of application data, such as in combination with Figures 2 to 8 As described in the example. For example, server 802 can generate at least one frame of application data, such as... Figure 8 As described in 810. Furthermore... Figure 1 The processing unit 120 can perform step 1002. In some aspects, generating application data for at least one frame may include rendering application data for at least one frame; for example, the apparatus may render application data for at least one frame. The application data may correspond to a set of values ​​or a multivariate function. The application data may be associated with a subset of at least one frame.

[0086] At 1004, the device can convert application data of at least one frame, which corresponds to one or more image functions or one or more data channels, such as in combination. Figures 2 to 8 As described in the example. For example, server 802 can convert at least one frame of application data, which corresponds to one or more image functions or one or more data channels, such as... Figure 8 As described in 812. Furthermore... Figure 1 The processing unit 120 can execute step 1004. One or more image functions may correspond to one or more two-dimensional (2D) image functions. Furthermore, each of the one or more image functions may include multiple pixels, and each of the multiple pixels may include one or more function values. The one or more function values ​​may include at least one of the following: one or more depth values, one or more textures, one or more values, texture coordinates, one or more transparency values, one or more screen masks, one or more normal vectors, one or more primitive identifiers (IDs), one or more global coordinate positions, or one or more shift maps.

[0087] At 1006, the device can encode application data for at least one frame, which is associated with a data stream and is encoded via video coding processing, such as in combination. Figures 2 to 8As described in the example. For example, server 802 can encode application data for at least one frame associated with a data stream, the application data being encoded via video encoding processing, such as... Figure 8 As described in 820. Furthermore... Figure 1 The processing unit 120 can perform step 1006. In some aspects, encoding application data for at least one frame can include subsampling the application data for at least one frame; for example, the apparatus can subsample the application data for at least one frame. The application data can be subsampled based on one or more subsampling channels, wherein at least one of the one or more subsampling channels can be a luminance (Y) chrominance (UV) (YUV) channel. Furthermore, the data stream can be a Moving Picture Experts Group (MPEG) stream or a video stream. Additionally, video encoding processing can include Moving Picture Experts Group (MPEG) encoding.

[0088] At 1008, the device can assign encoded application data to one or more video frames or one or more MPEG frames, such as in combination. Figures 2 to 8 As described in the example. For example, server 802 can assign encoded application data to one or more video frames or one or more MPEG frames, such as... Figure 8 As described in 830. Furthermore... Figure 1 The processing unit 120 can execute step 1008. The encoded application data can be allocated block by block for one or more video frames or can be allocated in parallel blocks.

[0089] At 1010, the device can store encoded application data in one or more video frames or one or more MPEG frames, such as in combination. Figures 2 to 8 As described in the example. For example, server 802 can store encoded application data in one or more video frames or one or more MPEG frames, such as... Figure 8 As described in 840. Furthermore... Figure 1 The processing unit 120 can execute step 1010.

[0090] At 1012, the device can divide one or more video frames into at least one sub-region associated with color data or at least one other sub-region associated with application data, such as in combination with... Figure 11 As described in the example. For example, server 802 can divide one or more video frames into at least one sub-region associated with color data or at least one other sub-region associated with application data, such as... Figures 2 to 8 As described in 842. Furthermore... Figure 8 The processing unit 120 can execute step 1012.

[0091] At 1014, the apparatus can transmit the encoded application data of the at least one frame, as described in connection with the examples in Figure 1 At 1014, the apparatus can transmit the encoded application data of the at least one frame, as described in connection with the examples in Figures 2 to 8 At 1014, the apparatus can transmit the encoded application data of the at least one frame, as described in connection with the examples in Figure 8 The processing unit 120 in FIG. 8 can perform step 1014. The application data can be red (R), green (G), blue (B) (RGB) color data before being converted, and the application data can be luminance (Y) chrominance (UV) (YUV) color data after being encoded.

[0092] Figure 1 An example flow diagram 1100 of an example method in accordance with one or more techniques of the present disclosure is shown. The method can be performed by an apparatus such as a server, a client device, a CPU, a GPU, or an apparatus for graphics processing.

[0093] At 1102, the apparatus can receive application data of at least one frame, the application data being associated with a data stream, as described in connection with the examples in Figures 2 to 8 At 1102, the apparatus can receive application data of at least one frame, the application data being associated with a data stream, as described in connection with the examples in Figure 8 At 1102, the apparatus can receive application data of at least one frame, the application data being associated with a data stream, as described in connection with the examples in Figure 1 The processing unit 120 in FIG. 8 can perform step 1102. The application data can correspond to a set of values or a multivariate function. Further, the application data can be associated with a subset of the at least one frame. The data stream can be a Motion Picture Experts Group (MPEG) stream.

[0094] At 1104, the apparatus can decode the application data of the at least one frame, the application data being decoded via a video decoding process, as described in connection with the examples in Figure 12 At 1104, the apparatus can decode the application data of the at least one frame, the application data being decoded via a video decoding process, as described in connection with the examples in Figures 2 to 8 At 1104, the apparatus can decode the application data of the at least one frame, the application data being decoded via a video decoding process, as described in connection with the examples in Figure 8 The processing unit 120 in FIG. 8 can perform step 1104. In some aspects, decoding the application data of the at least one frame can include inverse sub-sampling the application data of the at least one frame, e.g., the apparatus can inverse sub-sample the application data of the at least one frame. The application data can be inverse sub-sampled based on one or more sub-sampled channels, where at least one of the one or more sub-sampled channels can be a luminance (Y) chrominance (UV) (YUV) channel. The video decoding process can include Motion Picture Experts Group (MPEG) decoding.

[0095] In some aspects, the decoded application data may be allocated for one or more video frames. Furthermore, the decoded application data may be stored within one or more video frames. The decoded application data may be allocated block-by-block for one or more video frames or may be allocated in parallel blocks. One or more video frames may be divided into at least one sub-region associated with color data or at least one other sub-region associated with application data.

[0096] At 1106, the device can convert at least one frame of application data, the converted application data corresponding to one or more image functions or one or more data channels, such as in combination. Figure 1 As described in the example. For example, client device 804 can convert at least one frame of application data, the converted application data corresponding to one or more image functions or one or more data channels, such as... Figures 2 to 8 As described in 880. Furthermore... Figure 8 The processing unit 120 can execute step 1106. The application data may be luminance (Y) chromaticity (UV) (YUV) color data before decoding, and may be red (R), green (G), and blue (B) (RGB) color data after conversion. One or more image functions may correspond to one or more two-dimensional (2D) image functions. Furthermore, each of the one or more image functions may include multiple pixels, and each of the multiple pixels may include one or more function values. The one or more function values ​​may include at least one of the following: one or more depth values, one or more texture values, one or more texture coordinates, one or more transparency values, one or more screen masks, one or more normal vectors, one or more primitive identifiers (IDs), one or more global coordinate positions, or one or more shift maps.

[0097] Figure 1 An example flowchart 1200 is shown, illustrating an example method according to one or more techniques of this disclosure. This method can be executed by a device such as a server, client device, CPU, GPU, or a device for graphics processing.

[0098] At 1202, the device can receive at least one frame of application data associated with a data stream, such as in combination. Figures 2 to 8 As described in the example. For example, client device 804 can receive at least one frame of application data associated with a data stream, such as... Figure 8 As described in 860. Furthermore... Figure 1The processing unit 120 in the example of FIG. 12 can perform step 1202. The application data can correspond to a set of values or a multivariate function. Further, the application data can be associated with a subset of at least one frame. The data stream can be a Motion Picture Experts Group (MPEG) stream.

[0099] At 1204, the device can decode the application data for the at least one frame, the application data decoded via a video decoding process as described in connection with Figures 2 to 8 the example of FIG. 12. For example, the client device 804 can decode the application data for the at least one frame, the application data decoded via a video decoding process as described in connection with Figure 8 870 of the example of FIG. 12. Further, Figure 1 The processing unit 120 in the example of FIG. 12 can perform step 1204. In some aspects, decoding the application data for the at least one frame can include inverse subsampling the application data for the at least one frame, e.g., the device can inverse subsample the application data for the at least one frame. The application data can be inverse subsampled based on one or more subsampled channels, where at least one of the one or more subsampled channels can be a luminance (Y) chrominance (UV) (YUV) channel. The video decoding process can include Motion Picture Experts Group (MPEG) decoding.

[0100] In some aspects, the decoded application data can be assigned for one or more video frames. Further, the decoded application data can be stored in one or more video frames. The decoded application data can be assigned block-wise for one or more video frames or can be assigned in parallel blocks. The one or more video frames can be divided into at least one sub-region associated with color data or at least one other sub-region associated with application data.

[0101] At 1206, the device can transform the application data for the at least one frame, the transformed application data corresponding to one or more image functions or one or more data channels as described in connection with ​ the example of FIG. 12. For example, the client device 804 can transform the application data for the at least one frame, the transformed application data corresponding to one or more image functions or one or more data channels as described in connection with ​ 880 of the example of FIG. 12. Further, ​The processing unit 120 in the example of FIG. 12 can perform step 1206. The application data can be luminance (Y) chrominance (UV) (YUV) color data before being decoded, and the application data can be red (R), green (G), blue (B) (RGB) color data after being converted. The one or more image functions can correspond to one or more two-dimensional (2D) image functions. Further, each of the one or more image functions can include a plurality of pixels, and each of the plurality of pixels can include one or more function values. The one or more function values can include at least one of one or more depth values, one or more texture values, one or more texture coordinates, one or more transparency values, one or more screen masks, one or more normal vectors, one or more primitive identifiers (IDs), one or more global coordinate positions, or one or more displacement maps.

[0102] At 1208, the device can display the application data of the at least one frame at a display panel, as described in connection with the example of FIG. 12. For example, the client device 804 can display the application data of the at least one frame at a display panel, as described in connection with 890 of FIG. 8. ​ ​ Further, the processing unit 120 in the example of FIG. 12 can perform step 1208. ​

[0103] ​​In one configuration, a method or apparatus for graphics processing is provided. The apparatus can be a server, a client device, a CPU, a GPU, or some other processor that can perform graphics processing. In one aspect, the apparatus can be a processing unit 120 within a device 104, or some other hardware within the device 104 or another device. The apparatus (e.g., processing unit 120) can include means for converting application data for at least one frame, the application data corresponding to one or more image functions or one or more data channels; means for encoding the application data for at least one frame, the application data associated with a data stream, the application data encoded via a video encoding process; means for transmitting the encoded application data for at least one frame; means for assigning the encoded application data to one or more video frames or one or more MPEG frames; means for storing the encoded application data in one or more video frames or one or more MPEG frames; means for dividing one or more video frames into at least one sub-region associated with color data or at least one other sub-region associated with application data; and means for generating the application data for at least one frame. The apparatus (e.g., processing unit 120) can also include means for receiving the application data for at least one frame, the application data associated with a data stream; means for decoding the application data for at least one frame, the application data decoded via a video decoding process; means for converting the application data for at least one frame, the converted application data corresponding to one or more image functions or one or more data channels; and means for displaying the application data for at least one frame at a display panel.

[0104] The subject matter described herein can be implemented to realize one or more benefits or advantages. For example, the described graphics processing techniques can be used by a server, a client, a GPU, a CPU, or some other processor that can perform graphics processing to implement the split rendering techniques described herein. This can also be implemented at a lower cost compared to other graphics processing techniques. Moreover, the graphics processing techniques herein can improve or speed up data processing or operations. Furthermore, the graphics processing techniques herein can improve resource or data utilization and / or resource efficiency. Moreover, aspects of the disclosure can utilize split rendering processing that can utilize image space function transmission over streaming video.

[0105] In accordance with the present disclosure, the term "or" can be construed as "and / or" unless the context otherwise requires. Moreover, the features herein disclosed can be construed as having such implied meanings even if such language is not used, unless the context otherwise requires, while phrases such as "one or more" or "at least one" can have been used for certain features but not others herein disclosed.

[0106] In one or more examples, the functions described herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media can include computer data storage media or communication media including any medium that facilitates transfer of a computer program from one place to another. In this manner, computer-readable media can generally correspond to (1) tangible computer- readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that can be used to carry or store instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products can include computer-readable media.

[0107] The code can be implemented by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), arithmetic logic units (ALUs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor," as used herein can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, the techniques could be fully implemented in one or more circuits or logic elements.

[0108] The techniques of this disclosure can be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or ICs, e.g., a chip set. Various components, modules, or units are described herein as being stored in or on one or more computer-readable media, such as computer-readable storage media. Such computer-readable media can include, without limitation, non- transitory computer-readable media, such as non-transitory computer-readable storage media and communication media. The computer-readable storage media can include, without limitation, one or more of the following: a hard disk, a floppy disk, a magnetic disk, an optical disk, a CD-ROM, a Blu-ray® disc, a DVD, a flash memory, a phase change memory, other memory, solid state memory devices, or a computer database. The communication media can include, without limitation, a signal, a carrier wave, or a computer program propagated across or through a computer network and / or a carrier wave.

[0109] Various examples have been described. These and other examples are within the scope of the following claims.

[0110] The following aspects are illustrative only and can be combined with other aspects or teachings described herein without limitation.

[0111] Aspect 1 is an apparatus for graphics processing at a server, the apparatus comprising at least one processor coupled to a memory and configured to: transform application data for at least one frame, the application data corresponding to one or more image functions or one or more data channels; encode the application data for the at least one frame, the application data being associated with a data stream, the application data being encoded via a video encoding process; and transmit the encoded application data for the at least one frame.

[0112] Aspect 2 is the apparatus of Aspect 1, wherein to encode the application data for the at least one frame, the at least one processor is configured to subsample the application data for the at least one frame.

[0113] Aspect 3 is the apparatus of any of Aspects 1 and 2, wherein the application data is subsampled based on one or more subsampled channels, at least one of the one or more subsampled channels being a luminance (Y) chrominance (UV) (YUV) channel.

[0114] Aspect 4 is the apparatus of any of Aspects 1 to 3, wherein the data stream is a Moving Picture Experts Group (MPEG) stream or a video stream.

[0115] Aspect 5 is the apparatus of any of Aspects 1 to 4, wherein the at least one processor is further configured to: allocate the encoded application data to one or more video frames or one or more MPEG frames.

[0116] Aspect 6 is the apparatus of any of Aspects 1 to 5, wherein the at least one processor is further configured to: store the encoded application data in one or more video frames or one or more MPEG frames.

[0117] Aspect 7 is the apparatus of any of Aspects 1 to 6, wherein the encoded application data is allocated block-wise or parallel block-wise for one or more video frames.

[0118] Aspect 8 is the apparatus of any of Aspects 1 to 7, wherein the at least one processor is further configured to: divide the one or more video frames into at least one sub-region associated with color data or at least one other sub-region associated with the application data.

[0119] Aspect 9 is the apparatus of any of Aspects 1 to 8, wherein the application data corresponds to a set of values or a multivariate function.

[0120] Aspect 10 is the apparatus of any of aspects 1 to 9, wherein the application data is red (R), green (G), blue (B) (RGB) color data prior to being converted, and wherein the application data is luminance (Y) chrominance (UV) (YUV) color data after being encoded.

[0121] Aspect 11 is the apparatus of any of aspects 1 to 10, wherein the application data is associated with a subset of the at least one frame.

[0122] Aspect 12 is the apparatus of any of aspects 1 to 11, wherein the at least one processor is further configured to generate the application data for the at least one frame.

[0123] Aspect 13 is the apparatus of any of aspects 1 to 12, wherein to generate the application data for the at least one frame, the at least one processor is configured to render the application data for the at least one frame.

[0124] Aspect 14 is the apparatus of any of aspects 1 to 13, wherein the video encoding process comprises a Moving Picture Experts Group (MPEG) encoding.

[0125] Aspect 15 is the apparatus of any of aspects 1 to 14, further comprising a transceiver coupled to the at least one processor, wherein the one or more image functions correspond to one or more two-dimensional (2D) image functions.

[0126] Aspect 16 is the apparatus of any of aspects 1 to 15, wherein each of the one or more image functions comprises a plurality of pixels, and wherein each of the plurality of pixels comprises one or more function values.

[0127] Aspect 17 is the apparatus of any of aspects 1 to 16, wherein the one or more function values comprise at least one of: one or more depth values, one or more texture one or more values, texture coordinates, one or more transparency values, one or more screen masks, one or more normal vectors, one or more primitive identifiers (IDs), one or more global coordinate positions, or one or more displacement maps.

[0128] Aspect 18 is an apparatus for graphics processing at a client device, the apparatus comprising at least one processor coupled to a memory and configured to: receive application data for at least one frame, the application data being associated with a data stream; decode the application data for the at least one frame, the application data being decoded via a video decoding process; and convert the application data for the at least one frame, the converted application data corresponding to one or more image functions or one or more data channels.

[0129] Aspect 19 is the apparatus of Aspect 18, wherein to decode the application data of the at least one frame, the at least one processor is configured to inverse subsample the application data of the at least one frame.

[0130] Aspect 20 is the apparatus of any one of Aspects 18 to 19, wherein the application data is inverse subsampled based on one or more subsampled channels, at least one of the one or more subsampled channels being a luminance (Y) chrominance (UV) (YUV) channel.

[0131] Aspect 21 is the apparatus of any one of Aspects 18 to 20, wherein the data stream is a Moving Picture Experts Group (MPEG) stream.

[0132] Aspect 22 is the apparatus of any one of Aspects 18 to 21, wherein the decoded application data is assigned for one or more video frames.

[0133] Aspect 23 is the apparatus of any one of Aspects 18 to 22, wherein the decoded application data is stored in one or more video frames.

[0134] Aspect 24 is the apparatus of any one of Aspects 18 to 23, wherein the decoded application data is assigned block-wise or parallel block-wise for one or more video frames.

[0135] Aspect 25 is the apparatus of any one of Aspects 18 to 24, wherein the one or more video frames are divided into at least one sub-region associated with color data or at least one other sub-region associated with application data.

[0136] Aspect 26 is the apparatus of any one of Aspects 18 to 25, wherein the application data corresponds to a set of values or a multivariate function.

[0137] Aspect 27 is the apparatus of any one of Aspects 18 to 26, wherein the application data, prior to being decoded, is luminance (Y) chrominance (UV) (YUV) color data, and wherein the application data, after being converted, is red (R), green (G), blue (B) (RGB) color data.

[0138] Aspect 28 is the apparatus of any one of Aspects 18 to 27, wherein the application data is associated with a subset of the at least one frame.

[0139] Aspect 29 is the apparatus of any one of Aspects 18 to 28, wherein the at least one processor is further configured to: display the application data of the at least one frame at a display panel.

[0140] Aspect 30 is the apparatus of any one of Aspects 18 to 29, wherein the video decoding process comprises Moving Picture Experts Group (MPEG) decoding.

[0141] Aspect 31 is an apparatus of any one of aspects 18 to 30, further comprising a transceiver coupled to the at least one processor, wherein the one or more image functions correspond to one or more two-dimensional (2D) image functions.

[0142] Aspect 32 is an apparatus of any one of aspects 18 to 31, wherein each of the one or more image functions comprises a plurality of pixels, and wherein each of the plurality of pixels comprises one or more function values.

[0143] Aspect 33 is an apparatus of any one of aspects 18 to 32, wherein the one or more function values comprise at least one of: one or more depth values, one or more texture values, one or more texture coordinates, one or more transparency values, one or more screen masks, one or more normal vectors, one or more primitive identifiers (IDs), one or more global coordinate positions, or one or more displacement maps.

[0144] Aspect 34 is a method of wireless communication for implementing any of aspects 1 to 33.

[0145] Aspect 35 is an apparatus for wireless communication, comprising means for implementing any of aspects 1 to 33.

[0146] Aspect 36 is a computer readable medium having stored computer executable code, wherein the code, when executed by a processor, causes the processor to implement any of aspects 1 to 33.

Claims

1. An apparatus for graphics processing at a server, comprising: Memory; as well as At least one processor, coupled to the memory, is configured to: Convert auxiliary data of at least one frame, the auxiliary data being provided as an image spatial function or one or more data channels, wherein the auxiliary data is RGB color data (Red R, Green G, Blue B) before conversion, and wherein the auxiliary data is Y color data (Luminance Y, Chromaticity UV YUV) after encoding. Interweave the converted auxiliary data of at least one frame with regular YUV color data; The interleaved data of the at least one frame is encoded, the interleaved data being associated with a data stream, and the interleaved data being encoded via video coding processing; as well as Transmit the encoded interleaved data of the at least one frame; In order to encode the interleaved data of the at least one frame, the at least one processor is configured to subsample the converted auxiliary data of the at least one frame; The converted auxiliary data is subsampled based on one or more subsampled channels, at least one of which is a YUV channel.

2. The apparatus according to claim 1, wherein, The data stream is a Moving Picture Experts Group (MPA) MPEG stream or a video stream.

3. The apparatus according to claim 1, wherein, The at least one processor is further configured to: The encoded interleaved data is distributed to one or more video frames or one or more MPEG frames.

4. The apparatus according to claim 3, wherein, The at least one processor is further configured to: The encoded interleaved data is stored in one or more video frames or one or more MPEG frames.

5. The apparatus according to claim 3, wherein, The encoded interleaved data is allocated block-by-block or in parallel block-by-block for the one or more video frames.

6. The apparatus according to claim 3, wherein, The at least one processor is further configured to: The one or more video frames are divided into at least one sub-region associated with color data or at least one other sub-region associated with the auxiliary data.

7. The apparatus according to claim 1, wherein, The auxiliary data corresponds to a set of values ​​or a multivariate function.

8. The apparatus according to claim 1, wherein, The auxiliary data is associated with a subset of the at least one frame.

9. The apparatus according to claim 1, wherein, The at least one processor is further configured to: The auxiliary data for the at least one frame is generated.

10. The apparatus according to claim 9, wherein, In order to generate the auxiliary data for the at least one frame, the at least one processor is configured to render the auxiliary data for the at least one frame.

11. The apparatus according to claim 1, wherein, The video encoding process includes Moving Picture Experts Group (MPA) MPEG encoding.

12. The apparatus of claim 1, further comprising a transceiver coupled to the at least one processor, wherein, The one or more image space functions include one or more of transparency, depth, and high dynamic range index.

13. An apparatus for graphics processing at a client device, comprising: Memory; as well as At least one processor, coupled to the memory, is configured to: Receive at least one frame of encoded interleaved data, the encoded interleaved data being associated with a data stream and including auxiliary YUV color data interleaved with regular luminance Y chrominance UV YUV color data; The encoded interleaved data of the at least one frame is decoded, and the encoded interleaved data is decoded via video decoding processing; Deinterleaving is performed on the decoded auxiliary YUV color data and the regular YUV color data; as well as The decoded auxiliary data of the at least one frame is converted, the converted auxiliary data corresponding to one or more image spatial functions or one or more data channels, wherein the auxiliary data is YUV color data before being decoded, and wherein the auxiliary data is RGB color data (Red R, Green G, Blue B) after being converted; In order to decode the encoded interleaved data of the at least one frame, the at least one processor is configured to perform inverse subsampling on the encoded interleaved data of the at least one frame. The encoded interleaved data is inversely subsampled based on one or more subsampled channels, at least one of which is a YUV channel.

14. The apparatus according to claim 13, wherein, The data stream is a Moving Picture Experts Group (Moving Picture Experts Group) MPEG stream.

15. The apparatus according to claim 13, wherein, The decoded interleaved data is assigned to one or more video frames.

16. The apparatus according to claim 15, wherein, The decoded interlaced data is stored in the one or more video frames.

17. The apparatus according to claim 15, wherein, The decoded interleaved data is allocated block-by-block or in parallel block-by-block for the one or more video frames.

18. The apparatus according to claim 15, wherein, The one or more video frames are divided into at least one sub-region associated with color data or at least one other sub-region associated with the auxiliary data.

19. The apparatus according to claim 13, wherein, The auxiliary data corresponds to a set of values ​​or a multivariate function.

20. The apparatus according to claim 13, wherein, The auxiliary data is associated with a subset of the at least one frame.

21. The apparatus according to claim 13, wherein, The at least one processor is further configured to: The auxiliary data of the at least one frame is displayed on the display panel.

22. The apparatus according to claim 13, wherein, The video decoding process includes Moving Picture Experts Group (MPA) MPEG decoding.

23. The apparatus of claim 13, further comprising a transceiver coupled to the at least one processor, wherein, The one or more image space functions include one or more of transparency, depth, and high dynamic range index.

24. A method for graphics processing at a server, comprising: The auxiliary data of at least one frame is converted, the auxiliary data being provided as an image spatial function or one or more data channels, wherein the auxiliary data is red (R), green (G), and blue (BRGB) color data before conversion, and wherein the auxiliary data is luminance (Y) and chrominance (UV) color data after encoding. Interweave the converted auxiliary data of at least one frame with regular YUV color data; The interleaved data of the at least one frame is encoded, the interleaved data being associated with a data stream, and the interleaved data being encoded via video coding processing; as well as Transmit the encoded interleaved data of the at least one frame; Encoding the interleaved data of the at least one frame includes subsampling the converted auxiliary data of the at least one frame; The converted auxiliary data is subsampled based on one or more subsampled channels, at least one of which is a YUV channel.

25. A method for graphics processing at a client device, comprising: Receive at least one frame of encoded interleaved data, the encoded interleaved data being associated with a data stream and including auxiliary YUV color data interleaved with regular luminance Y chrominance UV YUV color data; The encoded interleaved data of the at least one frame is decoded, and the encoded interleaved data is decoded via video decoding processing; Deinterleaving is performed on the decoded auxiliary YUV color data and the regular YUV color data; as well as The decoded auxiliary data of the at least one frame is converted, the converted auxiliary data corresponding to one or more image spatial functions or one or more data channels, wherein the auxiliary data is YUV color data before being decoded, and wherein the auxiliary data is RGB color data (Red R, Green G, Blue B) after being converted; Decoding the encoded interleaved data of the at least one frame includes performing inverse subsampling on the encoded interleaved data of the at least one frame; The encoded interleaved data is inversely subsampled based on one or more subsampled channels, at least one of which is a YUV channel.

26. An apparatus for graphics processing at a server, the apparatus comprising components for performing the method according to claim 24.

27. An apparatus for graphics processing at a client device, the apparatus comprising components for performing the method according to claim 25.

28. A computer-readable medium having program code recorded thereon, wherein the program code is executable by one or more processors of a server to cause the processors to perform the method according to claim 24.

29. A computer-readable medium having program code recorded thereon, wherein the program code is executable by one or more processors of a client device to cause the processor to perform the method according to claim 25.

30. A computer program product comprising computer-readable instructions that, when executed by a processor, cause the processor to perform the method according to claim 24 or 25.

Images