High performance and low latency implementation of wavelet-based image compression schemes

Abstract

This disclosure provides systems, devices, apparatus, and methods, including computer programs encoded on a storage medium, for high performance and low latency implementation of wavelet-based image compression schemes. The processor may generate a set of wavelet coefficients for the image frame based on the DWT for each of a plurality of horizontal and vertical partitions of the image frame in a single channel. The processor may select a set of bits associated with the set of wavelet coefficients to remove based on an entropy coding process. The processor may serialize the set of wavelet coefficients, where the serialized set of wavelet coefficients does not include the set of bits. The processor may output, for the second device, a bitstream including the set of serialized wavelet coefficients.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 373,991, filed on August 30, 2022, entitled “HIGH-PERFORMANCE AND LOW-LATENCY IMPLEMENTATION OF A WAVELET-BASED IMAGE COMPRESSION SCHEME,” and U.S. Non-Provisional Patent Application Serial No. 18 / 449,613, filed on August 14, 2023, entitled “HIGH-PERFORMANCE AND LOW-LATENCY IMPLEMENTATION OF A WAVELET-BASED IMAGE COMPRESSION SCHEME,” the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to processing systems and, more particularly, to one or more techniques for graphics processing. Background Art

[0004] Computer computing devices typically perform graphics and / or display processing (e.g., using a graphics processing unit (GPU), a central processing unit (CPU), a display processor, etc.) to render and display visual content. Such computing devices may include, for example, computer workstations, mobile phones such as smart phones, embedded systems, personal computers, tablet computers, and video game consoles. The GPU is configured to execute a graphics processing pipeline including one or more processing stages that operate together to execute graphics processing commands and output frames. The central processing unit (CPU) can control the operation of the GPU by issuing one or more graphics processing commands to the GPU. Modern CPUs are typically capable of executing multiple applications in parallel, wherein each application may need to use the GPU during execution. The display processor is configured to convert digital information received from the CPU into analog values, and can issue commands to a display panel for displaying visual content. A device that provides content for visual presentation on a display may utilize a CPU, a GPU, and / or a display processor.

[0005] Current image compression schemes may not be suitable for wireless streaming of high-resolution images between two wireless devices in a high-performance, low-latency, and energy-efficient manner. Improved techniques for wireless streaming of high-resolution images between two wireless devices are needed. Summary of the invention

[0006] A simplified summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key or important elements of all aspects, nor to describe the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that is given later.

[0007] In one aspect of the present disclosure, a method, a computer-readable medium, and an apparatus for performing graphics processing at a first device are provided. The apparatus includes a memory; and a processor, the processor being coupled to the memory, and based on information stored in the memory, the processor being configured to: generate a set of wavelet coefficients for an image frame based on a discrete wavelet transform (DWT) for each of a plurality of horizontal and vertical partitions of the image frame in a single channel; select a set of bits associated with the set of wavelet coefficients to be removed based on an entropy decoding process; serialize the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include a set of bits; and, for a second device, output a bitstream including the set of serialized wavelet coefficients.

[0008] In one aspect of the present disclosure, a method, computer-readable medium, and apparatus for performing graphics processing at a second device are provided. The apparatus includes a memory; and a processor, the processor being coupled to the memory, and based on information stored in the memory, the processor being configured to: obtain a bitstream associated with an image frame associated with a first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame; deserialize the set of serialized wavelet coefficients, wherein the set of deserialized wavelet coefficients includes a set of additional bits corresponding to a set of wavelet coefficients associated with the set of serialized wavelet coefficients; reconstruct the image frame based on the set of deserialized wavelet coefficients for each of a plurality of horizontal and vertical partitions of the image frame in a single channel, the set of deserialized wavelet coefficients being based on an inverse discrete wavelet transform (iDWT); and output an indication of the reconstructed image frame.

[0009] To achieve the above and related purposes, one or more aspects include features fully described below and specifically pointed out in the claims. The following description and the accompanying drawings set forth in detail some illustrative features of one or more aspects. However, these features are only indicative of some of the various ways in which the principles of the various aspects can be employed, and this specification is intended to include all such aspects and their related objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram illustrating an example content generation system in accordance with one or more techniques of this disclosure.

[0011] Figure 2 An example graphics processor (eg, a graphics processing unit (GPU)) is shown, in accordance with one or more techniques of this disclosure.

[0012] Figure 3 An example display framework including a display processor and a display is shown, in accordance with one or more techniques of this disclosure.

[0013] Figure 4 is a diagram illustrating an encoding / decoding pipeline in accordance with one or more techniques of this disclosure.

[0014] Figure 5 is a diagram illustrating an encoding / decoding pipeline in accordance with one or more techniques of this disclosure.

[0015] Figure 6 is a diagram illustrating example aspects of a discrete wavelet transform (DWT), in accordance with one or more techniques of this disclosure.

[0016] Figure 7 is a diagram illustrating further example aspects of a DWT in accordance with one or more techniques of this disclosure.

[0017] Figure 8 is a diagram illustrating an example of a first DWT implementation in accordance with one or more techniques of this disclosure.

[0018] Fig. 9 is a diagram illustrating an example of a second DWT implementation in accordance with one or more techniques of this disclosure.

[0019] Fig.10 is a diagram illustrating an example of an entropy coding implementation in accordance with one or more techniques of this disclosure.

[0020] Fig.11 is a diagram showing an example of dividing an input texture into regions according to one or more techniques of this disclosure.

[0021] Fig.12 is a diagram showing an example of dividing a region into columns of coefficients and forming coding groups according to one or more techniques of this disclosure.

[0022] Fig.13 is a diagram illustrating an example of conversion to a sign and magnitude representation and calculation of the most significant bit (MSB) in accordance with one or more techniques of this disclosure.

[0023] Fig.14 is a diagram illustrating an example of calculating truncated bits in accordance with one or more techniques of this disclosure.

[0024] Fig.15is a diagram illustrating an example of serialization and deserialization in accordance with one or more techniques of this disclosure.

[0025] Fig.16 is a diagram illustrating example aspects of an inverse discrete wavelet transform (iDWT), in accordance with one or more techniques of this disclosure.

[0026] Fig.17 is a call flow diagram illustrating example communications between an encoder and a decoder in accordance with one or more techniques of this disclosure.

[0027] Fig.18 is a flow chart of an example method of graphics processing in accordance with one or more techniques of this disclosure.

[0028] Fig.19 is a flow chart of an example method of graphics processing in accordance with one or more techniques of this disclosure.

[0029] Fig. 20 is a flow chart of an example method of graphics processing in accordance with one or more techniques of this disclosure.

[0030] Fig.21 is a flowchart of an example method of graphics processing according to one or more techniques of the present disclosure DETAILED DESCRIPTION

[0031] Various aspects of the system, device, computer program product and method will be described more fully in the following reference figures. However, the present disclosure can be embodied in a variety of different forms, and should not be interpreted as being limited to any specific structure or function presented in the present disclosure. On the contrary, providing these aspects makes the present disclosure thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. Based on the teachings of the present disclosure, those skilled in the art should understand that the scope of the present disclosure is intended to cover any aspect of the system, device, computer program product and method of the present disclosure, whether it is implemented independently of other aspects of the present disclosure or implemented in combination with other aspects of the present disclosure. For example, any number of aspects set forth herein can be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such a device or method implemented using other structures, functions, or structures and functions that are replacements or supplements of the various aspects of the disclosure set forth in the text. Any aspect disclosed herein can be embodied by one or more elements of the claims.

[0032] Various aspects are described herein, but many variations and arrangements of these aspects fall within the scope of the present disclosure. Although some potential benefits and advantages of various aspects of the present disclosure are mentioned, the scope of the present disclosure is not intended to be limited to specific benefits, uses or objectives. On the contrary, various aspects of the present disclosure are intended to be widely applicable to different wireless technologies, system configurations, processing systems, networks and transmission protocols, some of which are described by way of example in the drawings and the following description. The specific embodiments and drawings are merely illustrative of the present disclosure and are not limiting. The scope of the present disclosure is defined by the appended claims and their equivalents.

[0033] Several aspects are presented with reference to various devices and methods. These devices and methods are described in the following detailed description and shown in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether these elements are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0034] For example, an element, or any part of an element, or any combination of multiple elements can be implemented as a "processing system" including one or more processors (which can also be referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on chip (SOCs), baseband processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware that are configured to support the various functions described in this disclosure. One or more processors in a processing system can execute software. Software can be broadly interpreted as instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software components, applications, software applications, software packages, routines, sub-examples, objects, executable files, threads of execution, processes, functions, etc., whether referred to as software, firmware, middleware, microcodes, hardware description languages, or other.

[0035] The term application may refer to software. As described herein, one or more technologies may refer to an application (e.g., software) configured to perform one or more functions. In such an example, the application may be stored in a memory (e.g., an on-chip memory of a processor, a system memory, or any other memory). The hardware described herein (e.g., a processor) may be configured to execute an application. For example, an application may be described as including code that, when executed by hardware, causes the hardware to perform one or more technologies described herein. For example, the hardware may access code from a memory and execute the code accessed from the memory to perform one or more technologies described herein. In some examples, components are identified in the present disclosure. In such an example, a component may be hardware, software, or a combination thereof. A component may be a separate component, or a subcomponent of a single component.

[0036] In one or more examples described herein, the functions described can be implemented in hardware, software, or any combination. If implemented in software, the functions can be stored or encoded on a computer-readable medium as one or more instructions or codes. Computer-readable media include computer storage media. Storage media can be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, a combination of computer-readable media of the above type, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.

[0037] As used herein, instances of the term "content" may refer to "graphic content," "images," and the like, regardless of whether such terms are used as adjectives, nouns, or other parts of speech. In some examples, the term "graphic content," as used herein, may refer to content produced by one or more processes of a graphics processing pipeline. In further examples, the term "graphic content," as used herein, may refer to content produced by a processing unit configured to perform graphics processing. In some examples, the term "graphic content," as used herein, may refer to content produced by a graphics processing unit.

[0038] Certain image compression schemes (e.g., Joint Photographic Experts Group eXtra Small (JPEG XS), such as JPEG XS High Profile 420.12, JPEG XS High Profile 444.12, JPEG XS High Profile 4444.12) may not be suitable for wireless streaming of high-resolution images between two wireless devices in a high-performance, low-latency, and energy-efficient manner. In an example, some image compression schemes may allow for accurate reproduction of images for streaming across wireless devices, but such image compression schemes may be associated with relatively high latency, which may affect the user experience, for example, when the images are from a video game. In another example, some image compression schemes may allow for streaming of images in a manner associated with relatively low latency, but such image compression schemes may not enable the images to be accurately reconstructed at the receiving device.

[0039] Various technologies related to a high-performance, low-latency, and energy-efficient wavelet-based image compression scheme on a pair of SOCs with wireless connections of heterogeneous cores are described herein. In an example, a first device (e.g., a first wireless device) generates a set of wavelet coefficients for an image frame based on a discrete wavelet transform (DWT) for each of a plurality of horizontal and vertical partitions of the image frame in a single channel. The first device selects a set of bits associated with the set of wavelet coefficients to be removed based on an entropy decoding process. The first device serializes the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include a set of bits. The first device outputs a bitstream including the set of serialized wavelet coefficients to a second device. In another example, a second device (e.g., a second wireless device) obtains a bitstream associated with an image frame associated with the first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame. The second device deserializes the set of serialized wavelet coefficients, wherein the deserialized set of wavelet coefficients includes a set of additional bits corresponding to the set of wavelet coefficients associated with the set of serialized wavelet coefficients. The second device reconstructs the image frame based on a set of deserialized wavelet coefficients for each of a plurality of horizontal and vertical partitions of the image frame in a single channel, the set of deserialized wavelet coefficients being based on an inverse discrete wavelet transform (iDWT). The second device outputs an indication of the reconstructed image frame. Relative to the aforementioned techniques, wireless streaming of high-resolution images between two wireless devices can be implemented in a high-performance and low-latency manner. In addition, the aforementioned techniques can be performed without dedicated hardware blocks. In some aspects, a wavelet-based encoder / decoder associated with JPEG XS is described herein, but with reduced computational costs compared to the computational costs associated with JPEG XS. Example differences between the encoder / decoder described herein and the JPEG XS encoder / decoder may include fewer bits of storage per component, 16-bit integer math (as opposed to 32-bit integer math), a single channel wavelet formulation, use of a single most significant bit (MSB) decoding method, and use of different band-to-region mapping.

[0040] The examples described herein may refer to the use and functionality of a graphics processing unit (GPU). As used herein, a GPU may be any type of graphics processor, and a graphics processor may be any type of processor designed or configured to process graphics content. For example, a graphics processor or GPU may be a dedicated electronic circuit designed to process graphics content. As an additional example, a graphics processor or GPU may be a general-purpose processor configured to process graphics content.

[0041] Figure 11 is a block diagram showing an example content generation system 100 configured to implement one or more techniques of the present disclosure. The content generation system 100 includes a device 104. The device 104 may include one or more components or circuits for performing various functions described herein. In some examples, one or more components of the device 104 may be components of a SoC. The device 104 may include one or more components configured to perform one or more techniques of the present disclosure. In the example shown, the device 104 may include a processing unit 120, a content encoder / decoder 122, and a system memory 124. In some aspects, the device 104 may include multiple components (e.g., a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131). The display 131 may refer to one or more displays 131. For example, the display 131 may include a single display or multiple displays, and the multiple displays may include a first display and a second display. The first display may be a left-eye display, and the second display may be a right-eye display. In some examples, the first display and the second display may receive different frames for presentation thereon. In other examples, the first display and the second display may receive the same frame for presentation thereon. In further examples, the results of the graphics processing may not be displayed on the device, for example, the first display and the second display may not receive any frames for presentation thereon. Alternatively, the frames or graphics processing results may be transmitted to another device. In some aspects, this may be referred to as split rendering.

[0042] The processing unit 120 may include an internal memory 121. The processing unit 120 may be configured to perform graphics processing using the graphics processing pipeline 107. The content encoder / decoder 122 may include an internal memory 123. In some examples, the device 104 may include a processor that may be configured to perform one or more display processing techniques on one or more frames generated by the processing unit 120 before the frames are displayed on one or more displays 131. Although the processor in the example content generation system 100 is configured as a display processor 127, it should be understood that the display processor 127 is an example of a processor and other types of processors, controllers, etc. may be used as a substitute for the display processor 127. The display processor 127 may be configured to perform display processing. For example, the display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by the processing unit 120. The one or more displays 131 may be configured to display or otherwise present the frames processed by the display processor 127. In some examples, one or more displays 131 may include one or more of the following: a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head-mounted display, or any other type of display device.

[0043] Memory external to the processing unit 120 and the content encoder / decoder 122 (e.g., system memory 124) may be accessible to the processing unit 120 and the content encoder / decoder 122. For example, the processing unit 120 and the content encoder / decoder 122 may be configured to read from and / or write to an external memory such as the system memory 124. The processing unit 120 may be communicatively coupled to the system memory 124 on a bus. In some examples, the processing unit 120 and the content encoder / decoder 122 may be communicatively coupled to the internal memory 121 through a bus or via a different connection.

[0044] The content encoder / decoder 122 may be configured to receive graphics content from any source, such as the system memory 124 and / or the communication interface 126. The system memory 124 may be configured to store the received encoded or decoded graphics content. The content encoder / decoder 122 may be configured to receive the encoded or decoded graphics content in the form of encoded pixel data, such as from the system memory 124 and / or the communication interface 126. The content encoder / decoder 122 may be configured to encode or decode any graphics content.

[0045] Internal memory 121 or system memory 124 may include one or more volatile or non-volatile memories or storage devices. In some examples, internal memory 121 or system memory 124 may include RAM, static random access memory (SRAM), dynamic random access memory (DRAM), erasable programmable ROM (EPROM), EEPROM, flash memory, magnetic data medium or optical storage medium, or any other type of memory. According to some examples, internal memory 121 or system memory 124 may be a non-temporary storage medium. The term "non-temporary" may indicate that the storage medium is not embodied in a carrier or propagated signal. However, the term "non-temporary" should not be interpreted as meaning that internal memory 121 or system memory 124 is non-removable or its content is static. As an example, system memory 124 can be removed from device 104 and moved to another device. As another example, system memory 124 may not be removable from device 104.

[0046] Processing unit 120 may be a CPU, GPU, GPGPU, or any other processing unit that may be configured to perform graphics processing. In some examples, processing unit 120 may be integrated into the motherboard of device 104. In some examples, processing unit 120 may be present on a graphics card installed in a port in the motherboard of device 104, or may be otherwise incorporated into a peripheral device configured to interoperate with device 104. Processing unit 120 may include one or more processors, for example, one or more microprocessors, GPUs, ASICs, FPGAs, arithmetic logic units (ALUs), DSPs, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, processing unit 120 may store instructions for software in a suitable non-temporary computer-readable storage medium (e.g., internal memory 121), and may use one or more processors to execute instructions in hardware to perform the technology of the present disclosure. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0047] The content encoder / decoder 122 may be any processing unit configured to perform content decoding. In some examples, the content encoder / decoder 122 may be integrated into the mainboard of the device 104. The content encoder / decoder 122 may include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination. If the technology is partially implemented in software, the content encoder / decoder 122 may store instructions for the software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 123), and may use one or more processors to execute instructions in hardware to perform the technology of the present disclosure. Any of the foregoing (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0048] In some aspects, the content generation system 100 may include a communication interface 126. The communication interface 126 may include a receiver 128 and a transmitter 130. The receiver 128 may be configured to perform any receiving functions described herein with respect to the device 104. In addition, the receiver 128 may be configured to receive information from another device, such as eye or head position information, rendering commands, and / or positioning information. The transmitter 130 may be configured to perform any sending functions described herein with respect to the device 104. For example, the transmitter 130 may be configured to send information (which may include a request for content) to another device. The receiver 128 and the transmitter 130 may be combined into a transceiver 132. In such examples, the transceiver 132 may be configured to perform any receiving functions and / or sending functions described herein with respect to the device 104.

[0049] Reference again Figure 1As shown, in some aspects, the processing unit 120 may include an encoder 198, which is configured to generate a set of wavelet coefficients for an image frame based on a discrete wavelet transform (DWT) for each of multiple horizontal and vertical partitions of the image frame in a single channel; based on an entropy decoding process, select a set of bits associated with the set of wavelet coefficients to be removed; serialize the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include the bit set; and, for a second device, output a bit stream including the set of serialized wavelet coefficients. In some aspects, the processing unit 120 may include a decoder 199 configured to obtain a bitstream associated with an image frame associated with a first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame; deserialize the set of serialized wavelet coefficients, wherein the set of deserialized wavelet coefficients includes a set of additional bits corresponding to a set of wavelet coefficients associated with the set of serialized wavelet coefficients; reconstruct the image frame based on the set of deserialized wavelet coefficients for each of a plurality of horizontal and vertical partitions of the image frame in a single channel, the set of deserialized wavelet coefficients being based on an inverse discrete wavelet transform (iDWT); and output an indication of the reconstructed image frame. Although the following description may focus on graphics processing, the concepts described herein may be applicable to other similar processing techniques. In one aspect, the encoder 198 and / or the decoder 199 may be or include a content encoder / decoder 122.

[0050] As described herein, a device such as device 104 may refer to any device, apparatus or system configured to perform one or more technologies described herein. For example, a device may be a server, a base station, a user device, a client device, a station, an access point, a computer (e.g., a personal computer, a desktop computer, a laptop computer, a tablet computer, a computer workstation or a mainframe computer), a terminal product, an apparatus, a phone, a smart phone, a server, a video game platform or console, a handheld device (e.g., a portable video game device or a personal digital assistant (PDA)), a wearable computing device (e.g., a smart watch, an augmented reality device or a virtual reality device), a non-wearable device, a display or display device, a television, a television set-top box, an intermediate network device, a digital media player, a video streaming device, a content streaming device, an onboard computer, any mobile device, any device configured to generate graphic content, or any device configured to perform one or more technologies described herein. The process herein may be described as being performed by a specific component (e.g., a GPU), but in other embodiments, other components (e.g., a CPU) consistent with the disclosed embodiments may be used to perform.

[0051] The GPU can process multiple types of data or data packets in the GPU pipeline. For example, in some aspects, the GPU can process two types of data or data packets, such as context register packets and draw call data. The context register packet can be a set of global state information, such as information about global registers, shaders, or constant data, which can adjust how the graphics context will be processed. For example, the context register packet may include information about the color format. In some aspects of the context register packet, there may be one or more bits indicating which workload belongs to the context register. In addition, multiple functions or programs can be run simultaneously and / or in parallel. For example, a function or program can describe a certain operation, such as a color mode or color format. Therefore, the context register can define multiple states of the GPU.

[0052] The context state can be used to determine how a single processing unit (e.g., a vertex fetcher (VFD), a vertex shader (VS), a shader processor, or a geometry processor) operates, and / or in which mode the processing unit operates. To do this, the GPU can use context registers and programming data. In some aspects, the GPU can generate workloads in the pipeline based on the context register definition of the mode or state, such as vertex or pixel workloads. Certain processing units (e.g., VFDs) can use these states to determine certain functions, such as how to assemble vertices. Since these modes or states may change, the GPU may need to change the corresponding context. In addition, the workload corresponding to the mode or state can follow the changing mode or state.

[0053] Figure 2 An example GPU 200 is illustrated in accordance with one or more techniques of this disclosure. Figure 2 As shown, GPU 200 includes a command processor (CP) 210, a draw call group 212, a VFD 220, a VS 222, a vertex cache (VPC) 224, a triangle setup engine (TSE) 226, a rasterizer (RAS) 228, a Z-pass engine (ZPE) 230, a pixel interpolator (PI) 232, a fragment shader (FS) 234, a rendering back end (RB) 236, an L2 cache (UCHE) 238, and a system memory 240. Although Figure 2 GPU 200 is shown to include processing units 220-238, but GPU 200 may include multiple additional processing units. In addition, processing units 220-238 are only examples, and any combination or order of processing units may be used by the GPU in accordance with the present disclosure. GPU 200 also includes command buffer 250, context register group 260, and context state 261.

[0054] like Figure 2 As shown, the GPU may parse the command buffer into context register packets (e.g., context register packets 260) and / or draw call data packets (e.g., draw call packets 212) using a CP (e.g., CP 210) or a hardware accelerator. Then, the CP 210 may send the context register packets 260 or the draw call data packets 212 to the processing units or blocks in the GPU through separate paths. Further, the command buffer 250 may alternate different states of context registers and draw calls. For example, the command buffer may simultaneously store the following information: context registers of context N, draw calls of context N, context registers of context N+1, and draw calls of context N+1.

[0055] The GPU can render images in a variety of different ways. In some cases, the GPU can use direct rendering and / or tile rendering to render images. In a tiled rendering GPU, the image can be divided or separated into different parts or tiles. After the image is divided, each part or tile can be rendered separately. The tiled rendering GPU can divide the computer graphics image into a grid format so that each part of the grid (i.e., tile) is rendered separately. In some aspects of tiled rendering, during the merging pass (binning pass), the image can be divided into different binary bits or tiles. In some aspects, during the merging pass, a visibility stream can be constructed, in which visible primitives or drawing calls can be identified. The rendering pass can be performed after the merging pass. In contrast to tiled rendering, direct rendering does not divide the frame into smaller binary bits or tiles. Instead, in direct rendering, the entire frame is rendered at once (i.e., there is no merging pass). In addition, some types of GPUs can allow both tiled rendering and direct rendering (e.g., flex rendering).

[0056] In some aspects, the GPU may apply a drawing or rendering process to different bins or tiles. For example, the GPU may render to one bin, and perform all drawing for the primitives or pixels in the bin. During the process of rendering to the bin, the rendering target may be located in the GPU internal memory (GMEM). In some instances, after rendering to one bin, the contents of the rendering target may be moved to the system memory, and the GMEM may be released to render the next bin. In addition, the GPU may render to another bin, and perform drawing for the primitives or pixels in the bin. Therefore, in some aspects, there may be a small number of bins (e.g., four bins) that cover all the drawing in a surface. In addition, the GPU may loop through all the drawing in a bin, but perform drawing for visible drawing calls, i.e., drawing calls containing visible geometry. In some aspects, a visibility stream may be generated, e.g., in a merge pass, to determine visibility information for each primitive in an image or scene. For example, such a visibility stream may identify whether a primitive is visible. In some aspects, this information can be used to remove invisible primitives so that the invisible primitives are not rendered, for example, during the rendering process. In addition, at least some primitives identified as visible can be rendered during the rendering process.

[0057] In some aspects of tiled rendering, there may be multiple processing stages or channels. For example, rendering may be performed in two channels, e.g., a visibility or binary bit visibility channel and a rendering or binary bit rendering channel. During the visibility channel, the GPU may input a rendering workload, record the position of a primitive or triangle, and then determine which primitives or triangles fall into which binary bit or region. In some aspects of the visibility channel, the GPU may also identify or mark the visibility of each primitive or triangle in the visibility stream. During the rendering channel, the GPU may input a visibility stream and process one binary bit or region at a time. In some aspects, the visibility stream may be analyzed to determine which primitives or primitive vertices are visible or invisible. Therefore, visible primitives or primitive vertices may be processed. By doing so, the GPU may reduce the unnecessary workload of processing or rendering invisible primitives or triangles.

[0058] In some aspects, during the visibility channel, certain types of primitive geometry may be processed, for example, position-only geometry. In addition, primitives may be classified into different bins or regions, depending on the position or positioning of the primitives or triangles. In some cases, the classification of primitives or triangles into different bins may be performed by determining visibility information for these primitives or triangles. For example, the GPU may determine visibility information for each primitive in each bin or region, or write it, for example, to a system memory. The visibility information may be used to determine or generate a visibility stream. In a rendering pass, the primitives in each bin may be rendered separately. In these cases, a visibility stream may be extracted from memory and used to remove primitives that are not visible to the bin.

[0059] Some aspects of the GPU or GPU architecture can provide multiple different options for rendering (e.g., software rendering and hardware rendering). In software rendering, the driver or CPU can copy the entire frame of geometry by processing each view at a time. In addition, some different states can change depending on the view. Therefore, in software rendering, the software can copy the entire workload by changing some states that can be used to render for each viewpoint in the image. In some aspects, since the GPU may submit the same workload multiple times for each viewpoint in the image, there may be an increased amount of overhead. In hardware rendering, the hardware or GPU may be responsible for copying or processing the geometry of each viewpoint in the image. Therefore, the hardware can manage the copying or processing of the primitives or triangles for each viewpoint in the image.

[0060] Figure 3 is a block diagram 300 illustrating an example display framework including processing unit 120 , system memory 124 , display processor 127 , and display 131 , as may be identified in connection with device 104 .

[0061] The GPU may be included in a device that provides content for visual presentation on a display. For example, the processing unit 120 may include a GPU 310 that is configured to render graphics data for display on a computing device (e.g., device 104), which may be a computer workstation, a mobile phone, a smart phone or other smart device, an embedded system, a personal computer, a tablet computer, a video game console, etc. The operation of the GPU 310 may be controlled based on one or more graphics processing commands provided by the CPU 315. The CPU 315 may be configured to execute multiple applications simultaneously. In some cases, each of the multiple applications executed simultaneously may utilize the GPU 310 simultaneously. The processing technique may be performed by outputting frames on a physical or wireless communication channel via the processing unit 120.

[0062] System memory 124, which may be executed by processing unit 120, may include user space 320 and kernel space 325. User space 320 (sometimes referred to as "application space") may include software applications and / or application frameworks. For example, software applications may include operating systems, media applications, graphics applications, workspace applications, etc. An application framework may include a framework used by one or more software applications, such as libraries, services (e.g., display services, input services, etc.), application program interfaces (APIs), etc. Kernel space 325 may also include a display driver 330. Display driver 330 may be configured to control display processor 127. For example, display driver 330 may cause display processor 127 to form a frame and send data for the frame to a display.

[0063] Display processor 127 includes display control block 335 and display interface 340. Display processor 127 may be configured to manipulate functions of display 131 (e.g., based on input received from display driver 330). Display control block 335 may also be configured to output image frames to display 131 via display interface 340. In some examples, display control block 335 may additionally or alternatively perform post-processing on the provided image data based on execution of processing unit 120 on system memory 124.

[0064] The display interface 340 may be configured to cause the display 131 to display an image frame. The display interface 340 may output image data to the display 131 according to an interface protocol (e.g., MIPIDSI (Mobile Industry Processor Interface, Display Serial Interface)). That is, the display 131 may be configured according to the MIPIDSI standard. The MIPIDSI standard supports a video mode and a command mode. In an example where the display 131 operates in a video mode, the display processor 127 may continuously refresh the graphics content of the display 131. For example, the entire graphics content may be refreshed at each refresh cycle (e.g., line by line). In an example where the display 131 operates in a command mode, the display processor 127 may write the graphics content of the frame into the buffer 350.

[0065] In some such examples, display processor 127 may not be able to continuously refresh the graphics content of display 131. Instead, display processor 127 may use vertical synchronization (Vsync) pulses to coordinate the rendering and consumption of graphics content at buffer 350. For example, when a Vsync pulse is generated, display processor 127 may output new graphics content to buffer 350. Thus, the generation of a Vsync pulse may indicate that current graphics content has been rendered at buffer 350.

[0066] The frame is displayed at the display 131 based on the display controller 345, the display client 355, and the buffer 350. The display controller 345 can receive image data from the display interface 340 and store the received image data in the buffer 350. In some examples, the display controller 345 can output the image data stored in the buffer 350 to the display client 355. Thus, the buffer 350 can represent a local memory to the display 131. In some examples, the display controller 345 can output the image data received from the display interface 340 directly to the display client 355.

[0067] The display client 355 may be associated with a touch panel that senses interactions between a user and the display 131. When a user interacts with the display 131, one or more sensors in the touch panel may output a signal to the display controller 345 indicating which of the one or more sensors has sensor activity, the duration of sensor activity, the pressure applied to the one or more sensors, etc. The display controller 345 may use the sensor output to determine the manner in which the user is interacting with the display 131. The display 131 may further be associated with / include other devices that operate in conjunction with the display client 355, such as a camera, a microphone, and / or a speaker.

[0068] Some processing techniques of device 104 may be performed in three stages (e.g., stage 1: rendering stage; stage 2: composition stage; and stage 3: display / transmission stage). However, other processing techniques may combine the composition stage and the display / transmission stage into a single stage, such that the processing technique may be performed based on two overall stages (e.g., stage 1: rendering stage; and stage 2: composition / display / transmission stage). During the rendering stage, GPU 310 may process a content buffer based on the execution of an application that generates content pixel by pixel. During the composition and display stages, the pixel elements may be assembled to form a frame that is transmitted to a physical display panel / subsystem (e.g., display 131) that displays the frame.

[0069] Instructions executed by a CPU (e.g., software instructions) or a display processor may cause the CPU or display processor to search and / or generate a composition strategy for synthesizing a frame based on dynamic priorities and runtime statistics associated with one or more synthesis strategy groups. A frame displayed by a physical display device such as a display panel may include multiple layers. Moreover, the synthesis of a frame may be based on combining multiple layers into a frame (e.g., based on a frame buffer). After combining multiple layers into a frame, the frame may be provided to a display panel for display thereon. The process of combining each layer in a plurality of layers into a frame may be referred to as synthesis, frame synthesis, synthesis process, synthesis process, etc.

[0070] The frame synthesis process or synthesis strategy can correspond to the technology for synthesizing different layers in multiple layers into a single frame. Multiple layers can be stored in a double data rate (DDR) memory. Each layer in multiple layers can further correspond to a separate buffer. A synthesizer or hardware synthesizer (HWC) associated with a block or function can determine the input of each layer / buffer, and perform a frame synthesis process to generate an output indicating a synthesized frame. That is, the input can be a layer and the output can be a frame synthesis process for synthesizing a frame to be displayed on a display panel.

[0071] Some aspects of display processing can utilize different types of mask layers, for example, shape mask layers. A mask layer is a layer that can represent a portion of a display or display panel. For example, an area of ​​a mask layer may correspond to an area of ​​a display, but the entire mask layer may depict a portion of what is actually displayed at the display or panel. For example, a mask layer may include the top and bottom of a display area, but the middle portion of the mask layer may be empty. In some examples, there may be multiple mask layers to represent different portions of a display area. Moreover, for certain portions of a display area, the contents of different mask layers may overlap with each other. Thus, a mask layer may represent a portion of a display area that may or may not overlap with other mask layers.

[0072] Figure 44 is a diagram 400 illustrating an encoding / decoding pipeline 402 according to one or more techniques of the present invention. The encoding / decoding pipeline 402 can facilitate wireless streaming of high-resolution images between two wireless devices in a high-performance, low-latency, and energy-efficient manner. The encoding / decoding pipeline 402 may include an encoder 404 and a decoder 406. In an example, the encoder 404 may be or include a first device 405 (e.g., a first wireless device) and the decoder 406 may be or include a second device 407 (e.g., a second wireless device). In an example, the first device 405 and / or the second device 407 may be the device 104. In an example, the first device 405 may be a video game console and the second device 407 may be a smartphone. In another example, the first device 405 may be a first video game console (e.g., a stationary video game console) and the second device 407 may be a second video game console (e.g., a handheld video game console), wherein the first video game console may have a first computing capability, and wherein the second video game console may have a second computing capability, wherein the first computing capability may be greater than the second computing capability. For example, the first video game console may be able to render graphical content at a higher frame rate, a higher resolution, etc., than graphical content rendered by the second video game console. In another example, the first device 405 may be a smartphone, and the second device 407 may be an extended reality (XR) headset, such as an augmented reality (AR) headset, a mixed reality (MR) headset, or a virtual reality (VR) headset. In a further example, the first device 405 may be a server and the second device 407 may be a handheld video game console. The first device 405 and the second device 407 may communicate with each other via a wired connection and / or a wireless connection. In an example, the wired connection may be or include an Ethernet connection and / or a universal serial bus (USB) connection. In an example, the wireless connection may be or include a 5G new radio (NR) connection, a Bluetooth® connection, or a wireless connection. TM (Bluetooth is a trademark of the Bluetooth Special Interest Group (SIG)) connection and / or wireless local area network (WLAN) connection, such as Wi-Fi based on the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard TM (Wi-Fi is a trademark of the Wi-Fi Alliance) connection.

[0073] The encoder 404 may include a first GPU 408 and / or a first CPU 410. The decoder 406 may include a second CPU 412, a digital signal processor (DSP) 414, and / or a second GPU 416. A digital signal processor (DSP) may refer to a dedicated microprocessor chip with an architecture designed for the operational characteristics of digital signal processing. In an example, the DSP 414 may be configured to perform an inverse discrete wavelet transform (iDWT). Although Figure 4 Not depicted, but encoder 404 may also include a DSP.

[0074] The encoder 404 may obtain (e.g., receive) a first frame 418. In an example, the first GPU 408 and / or the first CPU 410 may render the first frame 418. As used herein, the term frame may refer to a single image in a series of consecutive images to be displayed on a display panel. In an example, the first frame 418 may be a frame of a video game application.

[0075] The first GPU 408 of the encoder 404 may perform a discrete wavelet transform (DWT) 420 on the first frame 418. DWT may refer to a wavelet transform in which a wavelet is discretely sampled. Unlike other types of transforms (e.g., Fourier transforms) that capture frequency, DWT may capture both frequency and position information (i.e., time position). DWT may decompose a signal (e.g., a signal associated with an image) into wavelets. Wavelets may be obtained from a single prototype wavelet by scaling a single prototype wavelet according to a scaling parameter and moving the single prototype wavelet according to a moving parameter. Wavelets may refer to a set of basis functions for transforming data (e.g., pixels in the first frame 418). In one aspect, the first GPU 408 (or the first CPU 410) may divide the first frame 418 into slices 422 (e.g., four slices), and the first GPU 408 may perform DWT 420 on each of the slices 422 independently and / or in parallel. Slices may refer to the division of a frame. In an example, the slices 422 may be horizontal slices. In another example, the slice 422 may be a vertical slice. The output of the DWT may be a set of wavelet coefficients. The wavelet coefficients may refer to the convolution of data (eg, pixel values) with a wavelet basis.

[0076] The first GPU 408 of the encoder 404 may perform entropy decoding (EC) 424 on the output of the DWT 420. For example, the first GPU 408 may perform EC 424 on the output of the DWT 420 for each of the slices 422 independently and / or in parallel. EC (which may also be referred to as an entropy decoding process or an entropy encoding process) may refer to a lossless data compression method that attempts to approach the lower bound stated by Shannon's source decoding theorem, which states that a lossless data compression method may have an expected code length that is greater than or equal to the entropy of the source. EC 424 may calculate the number of bits associated with the first frame 418 (e.g., bits associated with the output of the DWT 420) that will be truncated (i.e., removed) as part of the compression process. The number of truncated bits may be associated with a lossy compression scheme.

[0077] The first CPU 410 of the encoder 404 may perform serialization 426 on the output of the EC 424. For example, the first CPU 410 may perform serialization 426 on the output of the EC 424 for each of the slices 422 independently and / or in parallel. Serialization may refer to the process of translating data (e.g., the output of the EC 424) into a format (e.g., a series of bytes) that can be stored or transmitted. Serialization 426 may include truncating the number of bits calculated by the EC 424.

[0078] At 428, the encoder 404 may send the output of the serialization 426 to the decoder 406 via a wired connection and / or a wireless connection. For example, the encoder 404 may send the output of the serialization 426 for each of the slices 422 independently and / or in parallel. In an example, sending the output of the serialization 426 may include generating a bitstream. A bitstream may refer to a sequence of bits. In one aspect, the encoder 404 may send the first packet after the first substream, which may enable deserialization to be performed in parallel (explained in more detail below).

[0079] The decoder 406 may obtain the output of the serialization 426 through a wired connection and / or a wireless connection. For example, the decoder 406 may obtain the output of the serialization 426 for each of the slices 520 independently and / or in parallel.

[0080] The second CPU 412 of the decoder 406 performs deserialization 430 independently and / or in parallel on the output of the serialization 426 through a wired connection and / or a wireless connection. For example, the second CPU 412 can perform deserialization 430 on the output of the serialization 426 of each of the slices 422. Deserialization can refer to the process of extracting a data structure from a series of bytes. Deserialization 430 can include adding bits to the output of the serialization 426, where the added bits can correspond to the truncated bits calculated by the EC 424. The output of the deserialization can include reconstructed wavelet coefficients.

[0081] The DSP 414 and / or the second GPU 416 of the decoder 406 may perform an inverse discrete wavelet transform (iDWT) 432 independently and / or in parallel on the output of the deserialization 430. In the example where the first frame 418 is divided into four slices, the DSP 414 may perform the iDWT 432 on the first slice of the four slices, and the second GPU 416 may perform the iDWT 432 on the second slice, the third slice, and the fourth slice of the four slices. Other partitions are also possible. For example, the DSP 414 may perform the iDWT 432 on the first slice and the second slice, and the second GPU 416 may perform the iDWT 432 on the third slice and the fourth slice. An inverse discrete wavelet transform (iDWT) may refer to the inversion of a DWT. The iDWT 432 may generate a first reconstructed frame 434, wherein the first reconstructed frame 434 may correspond to the first frame 418 (i.e., a version of the first frame 418). In an example, the time between the encoder 404 obtaining the frame and the decoder 406 generating the first reconstructed frame 434 can be approximately 13.8 ms plus additional time, which can depend on the input from the bridge chip source, the time for wireless transmission, and the time for display and vertical synchronization (vsync). The first reconstructed frame 434 can be rendered on the display panel.

[0082] The encoder 404 and the decoder 406 may perform processes similar to those described above on the second frame 436 and the third frame 438 to respectively generate a second reconstructed frame 440 and a third reconstructed frame 442. The first reconstructed frame 434, the second reconstructed frame 440, and the third reconstructed frame 442 may be sequentially rendered on a display panel as part of the video content.

[0083] Figure 55 is a diagram 500 illustrating an encoding / decoding pipeline 502 according to one or more techniques of the present disclosure. The encoding / decoding pipeline 502 may be or include the encoding / decoding pipeline 402. The encoding / decoding pipeline 502 may be or include an encoder 504 and a decoder 506. In an example, the encoder 504 may be or include the encoder 404. In an example, the decoder 506 may be or include the decoder 406. The encoder 504 may include a first CPU 508 and a first GPU 510. The encoder 504 may also include a DSP ( Figure 5 The decoder 506 may include a second CPU 512, a second GPU 514, and a DSP 516.

[0084] The encoder 504 may obtain an input image frame 518. The first CPU 508 may divide the input image frame 518 into slices 520. The GPU processing block 522 may perform a DWT 524 and an EC 526 on each of the slices 520. The output of the GPU processing block 522 may be placed in a CPU work queue 528 of the first CPU 508. The CPU processing block 530 may perform a bitstream generation (BG) 532 on the output in the CPU work queue 528. A network layer 534 associated with the first CPU 508 may generate a user datagram protocol (UDP) transmission block 536 based on the BG 532. The encoder 504 may send the UDP transmission block 536 to the decoder 506.

[0085] The decoder 506 may receive a UDP receive block 538, which may be or include a UDP send block 536. A network layer 540 associated with the second CPU 512 may place the UDP receive block 538 in a CPU work queue 542 of the second CPU 512. A CPU processing block 544 of the second CPU 512 may perform a BG 546 on the UDP receive block 538 (or a portion thereof) in the CPU work queue 542. The output of the BG 546 may be placed in a GPU work queue 548 of the second GPU 514. A GPU processing block 550 of the second GPU 514 may perform an iDWT 552 on the output of the BG 546 in the GPU work queue 548. The output of the BG 546 may also be placed in a DSP work queue 554 of the DSP 516. A DSP processing block 556 of the DSP 516 may perform an iDWT 558 on the output of the BG 546 in the DSP work queue 554. The outputs of the iDWT 552 and the iDWT 558 may be combined to generate an output image block 560. The output image block 560 may correspond to the first reconstructed frame 434. The output image block 560 may be presented on a display panel.

[0086] Figure 6408. FIG600 is a diagram illustrating example aspects of a discrete wavelet transform (DWT) in accordance with one or more techniques of this disclosure. The example aspects of the DWT may correspond to and / or be included in DWT 420. The example aspects of the DWT may be performed by and / or may be associated with first GPU 408.

[0087] The first GPU 408 can obtain an input image 602, wherein the input image 602 can be divided into four slices. The compute shader of the first GPU 408 can convert the input image 602 from a red-green-blue (RGB) format to a brightness blue-projection-red-projection (YUV) format. The RGB format can refer to a format for representing colors in which the red, green, and blue primary colors of light are added together to reproduce colors. The YUV format can refer to a format for representing colors that takes into account human perception, allowing the bandwidth of chromaticity components to be reduced compared to the RGB format. The computer shader can also perform DWT in a separate vertical channel and a separate horizontal channel. A compute shader can refer to a general-purpose shader that can calculate arbitrary information (e.g., graphics data, non-graphics data, etc.).

[0088] The first GPU 408 may perform the DWT using a highly configured LeGall 5 / 3 wavelet function, where "5" may refer to five vertical decomposition levels of the DWT and where "3" may refer to three horizontal decomposition levels of the DWT. The JPEG XS standard may also utilize the LeGall 5 / 3 wavelet function. The JPEG XS standard may refer to an interoperable, visually lossless, low-latency, and lightweight image and video coding system that targets mezzanine compression within audio-visual (AV) applications.

[0089] As described above, the compute shader of the first GPU 408 may perform the DWT in a separate vertical channel and a separate horizontal channel. Diagram 600 depicts a view of an input image 602 after two channels 604 (one horizontal channel and one vertical channel). As will be described in more detail below, the DWT for an image may include 5 horizontal channels and 3 vertical channels. In order to generate wavelet coefficients for vertical lines (corresponding to horizontal channels) or vertical lines (corresponding to vertical channels), the wavelet coefficients may be loaded and stored once while performing high-frequency and low-frequency calculations. In other words, the DWT may include multiple channels (e.g., 5 horizontal channels and 3 vertical channels), but the generation of wavelet coefficients may occur in a single channel within each of the multiple channels. Diagram 600 also depicts a view of the input image 602 after the DWT is completed 606. The compute shader of the first GPU 408 may perform the DWT separately for each of the four slices (i.e., four blocks) in order to facilitate parallelism with a CPU (e.g., the first CPU 410). The encoder (e.g., encoder 404) can select a number of slices into which the input image 602 will be divided in order to increase parallelism with the CPU. In an example, the number of slices is four. In one example, if the encoder selects too many slices, software overhead may be increased and there may be relatively small GPU dispatches. In another example, if the encoder selects too few slices, there may not be appropriate overlap with the CPU to promote parallelism.

[0090] Figure 7 408. FIG700 is a diagram illustrating further example aspects of a discrete wavelet transform (DWT) in accordance with one or more techniques of this disclosure. The example aspects of the DWT may correspond to and / or be included in DWT 420. The example aspects of the DWT may be performed by and / or may be associated with first GPU 408.

[0091] The first GPU 408 may perform DWT via dispatches (e.g., first dispatch 702, second dispatch 704, third dispatch 706, fourth dispatch 708, fifth dispatch 710, sixth dispatch 712, and seventh dispatch 714), as well as copy operations (e.g., first copy 716 and second copy 718). In the example, the first GPU 408 may calculate a thread group size of 128×4. The first dispatch 702 and the second dispatch 704 may operate on a full block / slice (2560*360 pixels). Subsequent pairs of dispatches (e.g., pairs of third dispatch 706, fourth dispatch 708, fifth dispatch 710, sixth dispatch 712, and seventh dispatch 714) may operate on 25% of the pixels of the previous pair.

[0092] The DWT performed by the first GPU 408 may differ from the DWT performed as part of the JPEG XS standard in the following aspects. First, the DWT performed by the first GPU 408 may utilize 10 bits of storage for each YUV component. In contrast, the DWT performed as part of the JPEG XS standard may utilize 20 bits of storage (20 bits of storage may be the nominal precision of the wavelet coefficients associated with the DWT). Second, the DWT performed by the first GPU 408 may utilize 16-bit integer math (i.e., 16-bit precision calculations). In contrast, the DWT performed as part of the JPEG XS standard may utilize 32-bit integer math. Third, the DWT performed by the first GPU 408 may not perform an initial scaling of the input image to 20 bits. In contrast, the DWT performed as part of the JPEG XS standard may scale the input image to 20 bits. Fourth, the DWT performed by the first GPU 408 may not perform a direct current (DC) input level shift. A DC input level shift may refer to changing the average amplitude of a waveform. In contrast, the DWT implemented as part of the JPEG XS standard may perform a DC input level shift to achieve a zero DC level in the input signal.

[0093] Figure 8 8 is a diagram 800 illustrating an example of a first DWT implementation 802 in accordance with one or more techniques of this disclosure. First DWT implementation 802 may be associated with DWT 420. First DWT implementation 802 may compute high channel and low channel wavelet coefficients due to two separate execution passes on output data from a previous pass.

[0094] According to the first DWT implementation 802, a GPU (e.g., the first GPU 408) can perform a first pass 804 and a second pass 806. The first pass 804 can operate on a one-dimensional (1D) input 808 (e.g., a row or column of pixels in an input image, such as the first frame 418). The 1D input 808 can include pixel values. During the first pass 804, the GPU can calculate a high-frequency value (H-value 810) based on the 1D input 808. The GPU can store the H-value 810 in an intermediate memory of the GPU. Storing the H-value 810 in the intermediate memory may be inefficient. More specifically, the H-value in the H-value 810 can be based on the difference between the first pixel value and the sum of the second pixel value and the third pixel value divided by 2, wherein the first pixel value, the second pixel value pixel value and the third pixel value correspond to the first pixel, the second pixel and the third pixel in the 1D input 808, and wherein the first pixel is located near the second pixel and the third pixel and is located between the second pixel and the third pixel. The following equations (I) and (II) respectively give the first H-value H 1 and the second H-value H2 Example calculations for .

[0095] (I)

[0096] (II)

[0097] The second pass 806 may operate on a 1D input 808 and an H-value 810. During the second pass 806, the GPU may calculate low frequency values ​​(i.e., L-values, in Figure 8 The output 812 of the second channel 806 may include the H-value 810 and the L-value. The output 812 may be or include a set of wavelet coefficients associated with the H-value 810 and the L-value. 2 An example calculation of is shown below in formula (III).

[0098] (III)

[0099] Fig. 9 900 is a diagram illustrating an example of a second DWT implementation 902 in accordance with one or more techniques of this disclosure. The second DWT implementation 902 can be associated with the DWT 420. The second DWT implementation 902 can be a single channel algorithm that does not utilize intermediate storage, and thus the second DWT implementation 902 can be more efficient than the first DWT implementation 802.

[0100] According to the second DWT implementation 902, a GPU (e.g., the first GPU 408) may execute a single channel 906. The single channel 906 may operate on a one-dimensional (1D) input 904 (e.g., rows or columns of pixels in an input image, such as the first frame 418). The 1D input 904 may include pixel values. The GPU may calculate the H-value in a manner similar to that described above in the description of the first DWT implementation 802. However, unlike the first DWT implementation 802, the second DWT implementation 902 may calculate the L-value based on the 1D input 904 (and not based on the H-value), that is, the calculation of the L-value may be independent of the calculation of the H-value. The L-value for L according to the second DWT implementation 902 is given below in Equation (IV): 2 Example calculations for .

[0101] (IV)

[0102] The output 908 of a single channel 906 may include an H-value and an L-value. The output 908 may be or include a set of wavelet coefficients associated with the H-value and the L-value.

[0103] Fig.101 is a diagram 1000 illustrating an example entropy coding implementation 1002 in accordance with one or more techniques of this disclosure. The entropy coding implementation 1002 may be performed by the first GPU 408. The entropy coding implementation 1002 may be included in and / or may correspond to the EC 424. The entropy coding implementation 1002 may be implemented in a compute shader.

[0104] Typically, entropy decoding (e.g., entropy decoding implementation 1002) can analyze wavelet coefficients to calculate the number of bits that can be truncated (i.e., removed) from the bits associated with the wavelet coefficients. Entropy decoding implementation 1002 can be implemented in a single compute shader at the first GPU 408. Entropy decoding implementation 1002 can divide the wavelet texture (i.e., the output of the DWT, such as the second DWT implementation 902) into regions, wherein each region can include multiple horizontal lines in the image. The example described in connection with entropy decoding implementation 1002 defines a region as four horizontal lines; however, it should be understood that a region can include different numbers (e.g., 2, 8, etc.) of horizontal lines. A budget can be allocated to each selection. Entropy decoding implementation 1002 can remove bits from the wavelet coefficients until the region "fits" within its budget. On the one hand, each region can have an assigned weight, wherein the weight is based on the importance of the region. For example, some regions may correspond to image regions that include more important data, and thus some regions may be assigned relatively higher weights than regions that include less important data. In an example, the entropy decoding implementation 1002 may generate frame sizes ranging from 1.1-1.2 megabytes (MB), such as 1.12MB, 1.15MB, 1.18MB, etc. The entropy decoding implementation 1002 may include / accommodate the wavelet coefficient texture into a buffer in preparation for CPU serialization.

[0105] The entropy decoding implementation 1002 may differ from the entropy decoding performed as part of the JPEG XS standard in the following aspects. First, in the entropy decoding implementation 1002, an exhaustive search for possible decoding techniques may not be performed. In contrast, the entropy decoding performed as part of the JPEG XS standard may include an exhaustive search for possible decoding techniques to optimize for maximum compression ratio. Second, in the entropy decoding implementation 1002, a single most significant bit (MSB) decoding method may be utilized. In contrast, the JPEG XS standard may utilize different MSB decoding methods. Third, in the entropy decoding implementation 1002, a refinement step may not be utilized. In contrast, the JPEG XS standard may involve calculating multiple candidate scene values ​​for each region through a refinement step. Fourth, the entropy decoding implementation 1002 may include a different band-to-region mapping compared to the band-to-region mapping in the entropy decoding associated with the JPEG XS standard. For example, in the entropy decoding implementation 1002, the regions may be filled with adjacent wavelet coefficients in the image space (i.e., each region may actually be a horizontal strip of the image after performing the DWT), whereas in the JPEG XS standard, regions may be constructed by copying wavelet coefficients from different wavelet bands. Fifth, the entropy decoding implementation 1002 may not utilize vertical prediction. In contrast, the JPEG XS standard may utilize vertical prediction as part of the entropy decoding process.

[0106] The entropy decoding implementation 1002 may operate on an input texture 1004, wherein the input texture 1004 may include and / or may be associated with wavelet coefficients 1006, wherein the wavelet coefficients may be generated by a DWT (e.g., DWT 420). The input texture 1004 may correspond to an image (e.g., a first frame 418). At 1008, the first GPU 408 may divide the input texture 1004 (and therefore, the wavelet coefficients 1006) into regions. At 1010, the first GPU 408 may divide each of the regions into columns of wavelet coefficients (e.g., 20 columns). At 1012, the first GPU 408 may iterate over the columns of each region to form a decoding group. At 1014, the first GPU 408 may convert each decoding group from a sign representation to a sign plus amplitude representation. At 1016, the first GPU 408 may calculate the MSB of each decoding group. At 1018, the first GPU 408 may iterate the valid scenes to calculate the number of truncated bits (i.e., bits to be removed). At 1020, the first GPU 408 may accumulate the footprints of all decoding groups in the region. At 1022, the first GPU 408 may find the maximum footprint that fits the region budget. At 1024, the first GPU 408 may write the scene value to memory. Certain aspects of the entropy decoding implementation process 1002 will be described in more detail below.

[0107] Fig.11 is a diagram 1100 showing an example 1102 of dividing an input texture into regions 1104 according to one or more techniques of this disclosure. Example 1102 may correspond to Fig.10 1008 in. As described above, the first GPU 408 can divide the input texture 1004 into regions 1104. Three wavelet coefficients can be assigned to each pixel associated with the texture. The regions can correspond to 4 horizontal lines and 2560 vertical lines in the input image associated with the input texture 1004. The first GPU 408 can assign a work group (having a size of 128×4) to each region in the regions 1104. In an example, the first GPU 408 can divide the input texture 1004 into 360 regions.

[0108] Fig.12 1200 is a diagram illustrating an example of dividing a region into columns of coefficients and forming coding groups according to one or more techniques of this disclosure. Diagram 1200 depicts a first example 1202 of dividing a region in region 1104 into columns 1204 (e.g., 20 columns) of wavelet coefficients. The first example 1202 may correspond to Fig.10 1010 in.

[0109] Diagram 1200 also depicts a second example 1206 of iterating columns and forming decoding groups. The second example 1206 may correspond to Fig.10 1012 in. The first GPU 408 may process texture data 1208 (e.g., input texture 1004), where the texture data 1208 may include a Y component 1210, a U component 1212, and a V component 1214 from a YUV color space. The first GPU 408 may iterate over column 1204 and form a decoding group 1216. A decoding group may refer to four adjacent wavelet coefficients from the same component (e.g., four Y components, four U components, or four V components).

[0110] Fig.13 is a diagram 1300 illustrating an example of conversion to a sign and magnitude representation and calculation of the most significant bit (MSB) in accordance with one or more techniques of this disclosure. Diagram 1300 depicts a first example 1302 of converting wavelet coefficients from a signed integer representation to a sign and magnitude representation. The first example 1302 may correspond to Fig.10 1014 in the decoding group 1304. The decoding group 1216 may include four Y components: Y 1 , Y 2 , Y 3 and Y 4, where each of the Y components is a signed integer representation (which may also be referred to as a two's complement representation). In a two's complement representation, the leftmost of the bit sequence may be a sign bit. In an example, when the sign bit is "0", the bit sequence may represent a positive number, and when the sign bit is "1", the bit sequence may represent a negative number. In an example, for the bit sequence "1000", "1" (i.e., the leftmost bit) may be a sign bit. In a two's complement representation, a negative representation of a number may be created by taking the two's complement representation of the corresponding positive number, flipping the bit (i.e., changing the "0" bit to a "1" bit, and changing the "1" bit to a "0" bit) and adding a value of 1. In an example, the number "+2" may be represented in a two's complement representation as "0010". The number "-2" may be represented in a two's complement representation as "1110".

[0111] The first GPU 408 may convert the decode group 1304 into a decode group 1306, wherein the decode group 1306 may be represented by a sign and magnitude, and wherein the decode group may include four Y components: Y 1 m , Y 2 m , Y 3 m and Y 4 m , where each of the Y components is represented in sign and magnitude. In the sign and magnitude representation, the leftmost bit of the bit sequence may be a sign bit. In an example, when the sign bit is "0", the bit sequence may represent a positive number, and when the sign bit is "1", the bit sequence may represent a negative number. The remaining bits in the bit sequence may represent a number in binary format. In an example, the number "+2" may be represented as "0010" in the sign and magnitude representation, and the number "-2" may be represented as "1010" in the sign and magnitude representation.

[0112] Diagram 1300 also depicts a second example 1308 of calculating the MSB of a decoding group (e.g., decoding group 1306). The MSB may refer to the bit with the largest value in a binary number (excluding the sign bit). The second example 1308 may correspond to Fig.10 1016 in. The first GPU 408 may calculate the MSB of the decoding group 1306 according to the following equation (V).

[0113] (V)

[0114] In equation (V), "|" may represent a bitwise OR operation. "Firstbithigh" may return the number of the position of the first non-zero bit in a bit sequence.

[0115] Fig.14 is a diagram 1400 illustrating an example 1402 of calculating truncated bits according to one or more techniques of this disclosure. Example 1402 may correspond to Fig.10 1018 in. The first GPU 408 may iterate all (valid) scenes 1404 to calculate the number of truncated bits (i.e., the number of bits to be truncated). A scene may refer to a preset that defines a truncation position in a decoding group (e.g., decoding group 1306) based on a wavelet frequency band associated with a wavelet coefficient. Iterating a (valid) scene may refer to iterating a truncation position. A scene may be valid if (1) the truncation position is not greater than the position of the MSB and (2) the number of bits to be truncated is not greater than the number of bits in the bit sequence.

[0116] In an example, a decoding group (e.g., decoding group 1306) may be represented by a bit sequence 1406, where the bit sequence 1406 may be "001010110010". The MSB 1416 of the bit sequence 1406 may be located at position 9 1414. There may be three valid scenarios for the bit sequence 1406: scenario 0, scenario 1, and scenario 2. In scenario 0, the truncation position for the bit sequence 1406 may be position 0 1408. If the bit sequence 1406 is truncated according to scenario 0, the first GPU 408 may truncate the bit at position 0 1408 and the first GPU 408 may retain the (MSB-1) bit. Therefore, the truncated bit sequence corresponding to scenario 0 may be "101011001" (9 bits in total). In scenario 1, the truncation position for the bit sequence 1406 may be position 3 1410. If the bit sequence 1406 is truncated according to scenario 1, the first GPU 408 may truncate the bit at position 3 1410, and the first GPU 408 may retain the (MSB-4) bits. Therefore, the truncated bit sequence corresponding to scenario 1 may be "101011" (6 bits in total). In scenario 2, the truncation position for the bit sequence 1406 may be position 6 1412. If the bit sequence 1406 is truncated according to scenario 2, the first GPU 408 may truncate the bit at position 6 1412, and the first GPU 408 may retain the (MSB-7) bits. Therefore, the truncated bit sequence corresponding to scenario 2 may be "101" (3 bits in total).

[0117] Return to reference Fig.10At 1020, the first GPU 408 may accumulate the footprints of all decoding groups (e.g., decoding group 1216) in a region (e.g., a region in region 1104). In an example, the first GPU 408 may add the footprints of all decoding groups in the region to calculate a total footprint. There may be a total value for each scene. The footprint for a scene may refer to the number of bits remaining for the decoding group in the region after truncating according to the scene.

[0118] At 1022, the first GPU 408 can find the maximum footprint that fits the budget. The budget can be determined by the target bit rate. The bit rate (e.g., the target bit rate) can refer to the number of bits to be sent for a frame. In an example, the target bit rate can be 1.0-2.0MB per frame, such as 1.5MB per frame, 1.6MB per frame, 1.7MB per frame, etc. The first GPU 408 can select the target bit rate based on various factors, such as network conditions, device capabilities, etc. At 1024, the first GPU 408 can write scene values ​​to memory. The scene value for a region can refer to the remaining bits of the region after truncation. In an example, for each region, a scene value can be written to memory.

[0119] Fig.15 is a diagram 1500 illustrating an example 1502 of serialization and deserialization according to one or more techniques of this disclosure. Example 1502 may correspond to Figure 4 The encoder 404 can generate wavelet coefficients 1504 via DWT, as described above in Figure 4-Figure 9 As described in the description.

[0120] The first CPU 410 of the encoder 404 may divide the wavelet coefficients 1504 into strips 1506. The first CPU 410 may further divide the strips into substreams 1508, wherein each of the substreams 1508 may include 60 lines (e.g., horizontal lines) corresponding to the input image (e.g., the first frame 418). The first CPU 410 (e.g., via a serializer) may generate a plurality of substreams 1508 of the same type as above. Figure 10-Figure 14 Describing the results calculated by the corresponding entropy decoding process (e.g., entropy decoding implementation 1002), bits are truncated from each of the substreams 1508. The first CPU 410 can independently serialize each of the substreams 1508 and send each of the substreams 1508 through a network 1510 (e.g., a wireless network, a wired network, etc.). The (serialized) substreams 1508 may include serialized wavelet coefficients having bits truncated therefrom as calculated by the entropy decoding process.

[0121] The decoder 406 may obtain (e.g., receive) the substreams 1508 via the network 1510. The second CPU 412 may independently receive and deserialize each of the substreams 1508. The second CPU 412 of the decoder 406 may combine the substreams 1508 into stripes 1512 corresponding to the stripes 1506. The second CPU 412 may combine the stripes 1512 to generate reconstructed wavelet coefficients 1514, wherein the reconstructed wavelet coefficients 1514 may correspond to the wavelet coefficients 1504. As part of generating the reconstructed wavelet coefficients 1514, the second CPU 412 (e.g., via a deserializer) may add bits (e.g., zero bits) to the wavelet coefficients of the stripe 1506, wherein the added bits correspond to the truncated bits calculated by the entropy decoding process.

[0122] Fig.16 is a diagram 1600 illustrating example aspects of an inverse discrete wavelet transform (iDWT), in accordance with one or more techniques of this disclosure. Fig.16 The example aspects in can be compared with Figure 4 The iDWT may be the iDWT 432 in the above description. Fig. 9 , that is, the iDWT may calculate a 1D output based on an input of an H-value and an L-value, wherein the 1D output may include values ​​of a row or column of pixels.

[0123] After performing deserialization, the decoder 406 can reconstruct the final image (e.g., the first reconstructed frame 434) from the wavelet coefficients. Compared with performing DWT, performing iDWT on the GPU can be more computationally intensive (e.g., memory intensive). Therefore, the decoder 406 can divide the workload of the iDWT between the DSP 414 and the second GPU 416. In an example, the DSP 414 can perform iDWT on the first slice of the wavelet coefficients, and the second GPU 416 can perform iDWT on the second slice of the wavelet coefficients, the third slice of the wavelet coefficients, and the fourth slice of the wavelet coefficients. Other divisions can also be used. For example, the DSP 414 can perform iDWT on the first slice of the wavelet coefficients and the second slice of the wavelet coefficients, and the second GPU 416 can perform iDWT on the third slice of the wavelet coefficients and the fourth slice of the wavelet coefficients. The output of the iDWT can be an image presented on a display panel (e.g., display 131).

[0124] like Fig.16As shown in , the decoder 406 can obtain input 1602. The input 1602 can be deserialized wavelet coefficients divided into four slices. The DSP 414 and the second GPU can perform iDWT on the input 1602. Figure 1600 depicts an intermediate view 1604 of the input 1602 before completing the iDWT. Figure 1600 also depicts a view of the input 1602 after the iDWT is completed 1606.

[0125] The processes described above performed by encoder 404 and decoder 406 (ie, modified JPEG XS) may differ from the encoding / decoding processes performed as part of the JPEG XS standard. Table 1 below details such differences.

[0126]

[0127]

[0128] Table 1 JPEG XS specification relative to improved JPEG XS

[0129] With respect to ID 1 in Table 1, skipping input data scaling and converting to YUV format can keep DWT / iDWT within 16-bit math and 10-bit storage. Skipping input data scaling and converting to YUV format can improve encoder (e.g., encoder 404) performance / power and may not have an observable quality impact on still images. With respect to ID 2 in Table 1, skipping DC level shifting can improve encoder performance / power and may not have an observable quality impact on still images.

[0130] Relative to ID 3 in Table 1, using 10-bit storage can minimize the amount of data transferred between the memory and the GPU (e.g., the first GPU 408). Using 10-bit storage can improve the performance / power of the encoder and can have minimal impact on the quality of still images. Relative to ID 4 in Table 1, using 16-bit integer precision can improve the power / performance of the encoder and can have minimal impact on the quality of still images. Relative to ID 5 in Table 1, performing calculations in a single channel can improve the power / performance of the encoder and may not have an impact on the quality of still images.

[0131] Relative to ID 6 in Table 1, using a predetermined set of coding techniques can improve the power / performance of the encoder and may not have an observable impact on the quality of still images in most scenes. Some impact may be observed in scenes with high-frequency image content (e.g., grass, forest, etc.). Relative to ID 7 in Table 1, calculating multiple candidate scene values ​​without a refinement step can improve the power / performance of the encoder and may not have an observable impact on the quality of still images in most scenes. Some impact may be observed in scenes with high-frequency image content (e.g., grass, forest, etc.). Relative to ID 8 in Table 1, filling regions with adjacent wavelet coefficients in image space can improve the power / performance of the encoder and may not have an observable impact on the quality of still images in most cases. Some impact may be observed in scenes with high-frequency image content (e.g., grass, forest, etc.). Filling regions with adjacent wavelet coefficients in image space may be more suitable for the memory access pattern of the GPU than constructing regions by copying coefficients from different wavelet bands.

[0132] Relative to ID 9 in Table 1, using 10-bit storage can minimize the amount of data transferred between the memory and the GPU (e.g., the second GPU 416). Using 10-bit storage can improve the performance / power of the decoder (e.g., the decoder 406) and can have minimal impact on the quality of still images. Relative to ID 10 in Table 1, using 16-bit integer precision can improve the power / performance of the decoder and can have minimal impact on the quality of still images. Relative to ID 11 in Table 1, generating samples in a single pass can improve the power / performance of the decoder and may have no impact on the quality of still images.

[0133] With respect to ID 12 in Table 1, skipping DC level shifting may improve the power / performance of the decoder and may have no observable impact on the quality of still images. In addition, the decoder may skip DC level shifting because the encoder may not perform DC level shifting on the input data.

[0134] In one aspect, graphical user interface (GUI) rendering (which may also be referred to as user interface (UI) rendering) may be embedded in the last iDWT channel (e.g., the last iDWT channel associated with iDWT 432). UI may refer to an interface that enables a user to interact with an electronic device through graphical icons. The rendering of the GUI may be written directly to the swap chain. The swap chain may refer to a series of virtual frame buffers used by a GPU (e.g., second GPU 416) and a graphics application programming interface (API) for frame rate stabilization, stutter reduction, and other purposes.

[0135] In one aspect, an encoder (e.g., encoder 404) may operate on a YUV texture. The encoder may apply a per-plane DWT. Each plane may use a separate dispatch. Each plane may have 16 bits of per-texel storage. In an example, each slice may have 7 x 3 compute shader dispatches.

[0136] In one aspect, the output of the iDWT may be in a YUV format. In such an aspect, a decoder (e.g., decoder 406) may (1) amplify the output of the iDWT, (2) sharpen the output of the iDWT, and (3) convert the output from the YUV format to an RGB format. In another aspect, the output of the iDWT may be in an RGB format. In such an aspect, a decoder (e.g., decoder 406) may perform hardware amplification and sharpening, wherein the hardware amplification and sharpening may be implemented in a DPU.

[0137] Fig.17 1700 is a call flow diagram illustrating example communications between an encoder 1702 and a decoder 1704 according to one or more techniques of the present disclosure. In an example, the encoder 1702 may be or include the encoder 404 or the encoder 504. In an example, the decoder 1704 may be the decoder 406 or the decoder 506. In an example, the encoder 1702 and / or the decoder 1704 may be the device 104 or include the device 104. In an example, the encoder 1702 may be included in the first device 405, and the decoder 1704 may be included in the second device 407. In an example, the encoder 1702 may include a CPU and / or a GPU. The encoder 1702 may also include a DSP. In an example, the decoder 1704 may include a CPU, a GPU, and / or a DSP.

[0138] At 1712, the encoder 1702 may generate a set of wavelet coefficients for the image frame based on a discrete wavelet transform (DWT) for each of a plurality of horizontal and vertical partitions of the image frame in a single channel. At 1714, the encoder 1702 may select a set of bits associated with the set of wavelet coefficients to be removed based on an entropy decoding process. At 1716, the encoder 1702 may serialize the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include a set of bits. At 1718, the encoder 1702 may output a bitstream including the set of serialized wavelet coefficients for a second device.

[0139] At 1710, the encoder 1702 may divide the image frame into a plurality of image slices, wherein generating a set of wavelet coefficients for the image frame at 1712 may include performing a plurality of DWTs on each of the plurality of image slices, respectively. At 1706, the encoder 1702 may obtain the image frame before generating the set of wavelet coefficients at 1712. At 1708, the encoder 1702 may convert the image frame from a red-green-blue (RGB) format to a luminance-chrominance (YUV) format before generating the set of wavelet coefficients at 1712 and after obtaining the image frame at 1706.

[0140] At 1720, the decoder 1704 may obtain a bitstream associated with an image frame associated with the first device, wherein the bitstream may include a set of serialized wavelet coefficients associated with the image frame. At 1722, the decoder 1704 may deserialize the set of serialized wavelet coefficients, wherein the set of deserialized wavelet coefficients may include a set of additional bits corresponding to the set of wavelet coefficients associated with the set of serialized wavelet coefficients. At 1724, the decoder 1704 may reconstruct the image frame based on the set of deserialized wavelet coefficients for each of a plurality of horizontal and vertical partitions of the image frame in a single channel, wherein the set of deserialized wavelet coefficients is based on an inverse discrete wavelet transform (iDWT). At 1726, the decoder 1704 may output an indication of the reconstructed image frame. At 1728, the decoder 1704 may render a user interface (UI) associated with the image frame based on the reconstructed image frame, wherein the rendering of the UI utilizes an on-chip graphics processor memory. At 1730 , the decoder 1704 may display the reconstructed image frame.

[0141] Fig.18 1800 is a flowchart of an example method of graphics processing according to one or more techniques of the present disclosure. The method can be performed by an apparatus, such as an apparatus for graphics processing, a GPU, a CPU, a DSP, a first device including an encoder (e.g., a first device 405 including encoder 404), a wireless communication device, etc. Figure 1-Figure 17 The method may be associated with various advantages at the device, such as facilitating wireless streaming of high-resolution images between two wireless devices in a high-performance, low-latency, and energy-efficient manner. In an example, the method may be performed by the encoder 198.

[0142] At 1802, a first device generates a set of wavelet coefficients of an image frame based on a discrete wavelet transform (DWT) for each of a plurality of horizontal and vertical partitions of the image frame in a single channel. For example, Fig.17At 1712, it is shown that the encoder 1702 can generate a set of wavelet coefficients for the image frame based on a discrete wavelet transform (DWT) for each of multiple horizontal and vertical partitions of the image frame in a single channel. In an example, the DWT can be or include DWT 420 and / or DWT 524. In an example, the DWT can include the above combined Figure 6-Figure 7 and Fig. 9 In an example, the DWT may be or include the second DWT implementation 902, and the single channel may be the single channel 906. In an example, the set of wavelet coefficients may be the output 908. In an example, the set of wavelet coefficients may be or include the wavelet coefficients 1006. In an example, the image frame may be or include the first frame 418, the second frame 436, the third frame 438, the input image frame 518, and / or the input image 602. In an example, 1802 may be performed by the encoder 198.

[0143] At 1804, the first device selects a set of bits associated with a set of wavelet coefficients to be removed based on an entropy decoding process. Fig.17 At 1714, it is shown that the encoder 1702 can select a set of bits associated with the set of wavelet coefficients to be removed based on an entropy decoding process. In an example, the entropy decoding process can correspond to EC 424. In an example, the entropy decoding process can be or include the entropy decoding implementation 1002. For example, the entropy decoding process can include the above combined Figure 10-Figure 14 In an example, the set of bits can be associated with Fig.14 In the example, 1804 may be performed by the encoder 198 .

[0144] At 1806, the first device serializes the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include the set of bits. Fig.17 At 1716, the encoder 1702 may serialize the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include the set of bits. In one example, serializing the set of wavelet coefficients may include serializing 426. In one example, serializing the set of wavelet coefficients may include the above combined Fig.15 For example, serializing the set of wavelet coefficients may correspond to the serialized wavelet coefficients 1504. In an example, 1806 may be performed by the encoder 198.

[0145] At 1808, the first device outputs a bitstream including the set of serialized wavelet coefficients to the second device. Fig.17At 1718, the encoder 1702 may output a bitstream including a set of serialized wavelet coefficients for a second device. In an example, the second device may be the second device 407. In an example, the second device may include a decoder 1704. In an example, Fig.15 The encoder 404 is shown to output a bitstream including a set of serialized wavelet coefficients for the second device. In an example, the bitstream may correspond to the substream 1508 or the transmission at 428. In an example, the bitstream may correspond to BG 532. In an example, 1808 may be performed by the encoder 198.

[0146] Fig.19 1900 is a flowchart of an example method of display processing according to one or more techniques of the present disclosure. The method can be performed by an apparatus, such as an apparatus for graphics processing, a GPU, a CPU, a DSP, a first device including an encoder (e.g., a first device 405 including encoder 404), a wireless communication device, etc., as described above. Figure 1-Figure 17 The method may be associated with various advantages at the device, such as facilitating wireless streaming of high-resolution images between two wireless devices in a high-performance, low-latency, and energy-efficient manner. In an example, the method (including the various aspects described in detail below) may be performed by the encoder 198.

[0147] At 1908, the first device generates a set of wavelet coefficients for the image frame based on a discrete wavelet transform (DWT) for each of a plurality of horizontal and vertical partitions of the image frame in a single channel. For example, Fig.17 At 1712, it is shown that the encoder 1702 can generate a set of wavelet coefficients for the image frame based on a discrete wavelet transform (DWT) for each of the multiple horizontal and vertical partitions of the image frame in a single channel. In an example, the DWT can be or include the DWT 420 and / or the DWT 524. In an example, the DWT can include the above combined Figure 6-Figure 7 and Fig. 9 In an example, the DWT may be or include the second DWT implementation 902, and the single channel may be the single channel 906. In an example, the set of wavelet coefficients may be the output 908. In an example, the set of wavelet coefficients may be or include the wavelet coefficients 1006. In an example, the image frame may be or include the first frame 418, the second frame 436, the third frame 438, the input image frame 518, and / or the input image 602. In an example, 1908 may be performed by the encoder 198.

[0148] At 1910, the first device selects a set of bits associated with a set of wavelet coefficients to be removed based on an entropy decoding process. Fig.17At 1714, it is shown that the encoder 1702 can select a set of bits associated with a set of wavelet coefficients to be removed based on an entropy decoding process. In an example, the entropy decoding process can correspond to EC 424. In an example, the entropy decoding process can be or include the entropy decoding implementation 1002. For example, the entropy decoding process can include the above combined Figure 10-Figure 14 In an example, the set of bits can be associated with Fig.14 In the example, 1910 may be performed by the encoder 198 .

[0149] At 1926, the first device serializes the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include the set of bits. Fig.17 At 1716, the encoder 1702 may serialize the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include the set of bits. In one example, the serialized set of wavelet coefficients may include serialization 426. In one example, the serialized set of wavelet coefficients may include the above combined Fig.15 For example, the set of serialized wavelet coefficients may correspond to the serialized wavelet coefficients 1504. In an example, 1926 may be performed by the encoder 198.

[0150] At 1934, the first device outputs a bitstream including a set of serialized wavelet coefficients to the second device. Fig.17 At 1718, the encoder 1702 may output a bitstream including a set of serialized wavelet coefficients for a second device. In an example, the second device may be the second device 407. In an example, the second device may include a decoder 1704. In an example, Fig.15 The encoder 404 is shown to output a bitstream including a set of serialized wavelet coefficients for the second device. In an example, the bitstream may correspond to the substream 1508 or the transmission at 428. In an example, the bitstream may correspond to BG 532. In an example, 1934 may be performed by the encoder 198.

[0151] In one aspect, at 1906, the first device may divide the image frame into a plurality of image slices, wherein generating a set of wavelet coefficients for the image frame may include performing a plurality of DWTs on each of the plurality of image slices, respectively. Fig.17At 1710, the encoder 1702 may divide the image frame into a plurality of image slices, wherein generating a set of wavelet coefficients for the image frame at 1712 may include performing a plurality of DWTs on each of the plurality of image slices, respectively. In an example, the plurality of image slices may be or include slices 422. In an example, dividing the image frame into the plurality of image slices may correspond to the input image 602. In an example, 1906 may be performed by the encoder 198.

[0152] In one aspect, at 1902, the first device may obtain an image frame before generating a set of wavelet coefficients. For example, Fig.17 At 1706, the encoder 1702 may obtain an image frame before generating a set of wavelet coefficients at 1712. In another example, Figure 4 It is shown that the first device 405 can obtain the first frame 418. In an example, 1902 can be performed by the encoder 198.

[0153] In one aspect, at 1904, the first device may convert the image frame from a red-green-blue (RGB) format to a luminance-chrominance (YUV) format before generating the set of wavelet coefficients and after obtaining the image frame. For example, Fig.17 At 1708, encoder 1702 is shown to convert the image frame from a red-green-blue (RGB) format to a luminance-chrominance (YUV) format before generating a set of wavelet coefficients at 1712 and after obtaining the image frame at 1706. In an example, 1904 may be performed by encoder 198.

[0154] In one aspect, generating a set of wavelet coefficients may include generating the set of wavelet coefficients in a single channel without intermediate storage. For example, generating a set of wavelet coefficients at 1712 may include generating a set of wavelet coefficients in a single channel without intermediate storage. In addition, Fig. 9 It is also shown that generating the set of wavelet coefficients may include generating the set of wavelet coefficients in a single channel without intermediate storage.

[0155] In one aspect, generating a set of wavelet coefficients may include performing a DWT on the image frame based on one or more 16-bit precision calculations to generate a set of wavelet coefficients. For example, generating a set of wavelet coefficients at 1712 may include performing a DWT on the image frame based on one or more 16-bit precision calculations to generate a set of wavelet coefficients.

[0156] In one aspect, generating a set of wavelet coefficients may include assigning the set of wavelet coefficients to a 12-10-10 bit storage format based on one or more 16-bit precision calculations. For example, generating a set of wavelet coefficients at 1712 may include assigning the set of wavelet coefficients to a 12-10-10 bit storage format based on one or more 16-bit precision calculations.

[0157] In one aspect, at 1912, a set of bits of a set of wavelet coefficients to be removed is selected based on an entropy decoding process, which may include dividing the set of wavelet coefficients into regions, wherein each of the regions may be associated with a plurality of horizontal lines associated with an image frame. For example, at 1714, selecting a set of bits of a set of wavelet coefficients to be removed based on an entropy decoding process may include dividing the set of wavelet coefficients into regions, wherein each of the plurality of regions may be associated with a plurality of horizontal lines associated with an image frame. The foregoing aspects may correspond to 1008 and / or example 1102. In an example, the region may be or include region 1104. In an example, 1912 may be performed by encoder 198.

[0158] In one aspect, at 1914, a set of bits in a set of wavelet coefficients to be removed is selected based on an entropy decoding process, and the entropy decoding process may include: dividing a region in a plurality of regions into columns, wherein the set of bits selected may be based on at least one of the region or the column. For example, selecting a set of bits in a set of wavelet coefficients to be removed based on an entropy decoding process at 1714 may include: dividing a region in a plurality of regions into columns, wherein the set of bits selected may be based on at least one of the region or the column. The foregoing aspects may correspond to 1010 and / or the first example 1202. For example, a column may be or include column 1204. In an example, 1914 may be performed by encoder 198.

[0159] In one aspect, at 1916, selecting a set of bits in a set of wavelet coefficients to be removed based on an entropy decoding process may also include: forming a decoding group for the region, wherein the decoding group for the region may include: a plurality of adjacent wavelet coefficients associated with the same luminance or brightness component of the image frame. For example, selecting a set of bits in a set of wavelet coefficients to be removed based on an entropy decoding process at 1714 may also include: forming a decoding group for the region, wherein the decoding group for the region may include a plurality of adjacent wavelet coefficients associated with the same luminance or brightness component of the image frame. The foregoing aspects may correspond to 1012 and / or the second example 1206. For example, the decoding group may be or include the decoding group 1216. The luminance or brightness component may be or include the Y component 1210, the U component 1212, or the V component 1214. In an example, 1916 may be performed by the encoder 198.

[0160] In one aspect, at 1918, selecting a set of bits of a set of wavelet coefficients to be removed based on an entropy decoding process may also include converting a decoding group from a signed integer format to a sign and magnitude integer format. For example, selecting a set of bits of a set of wavelet coefficients to be removed based on an entropy decoding process at 1714 may also include converting a decoding group from a signed integer format to a sign and magnitude integer format. The foregoing aspects may correspond to 1016 and / or the first example 1302. For example, the decoding group (in a signed integer format) may be or include the decoding group 1304, and the decoding group (in a sign and magnitude integer format) may be or include the decoding group 1306. In an example, 1918 may be performed by the encoder 198.

[0161] In one aspect, at 1920, selecting a set of bits of a set of wavelet coefficients to be removed based on an entropy decoding process may also include: determining a most significant bit (MSB) of a decoding group, wherein the selection of the bit set may be based on the MSB of the decoding group. For example, at 1714, selecting a set of bits of a set of wavelet coefficients to be removed based on an entropy decoding process may also include: determining a most significant bit (MSB) of a decoding group, wherein the selection of the bit set may be based on the MSB of the decoding group. The foregoing aspects may correspond to 1016 and / or the second example 1308. In an example, 1920 may be performed by an encoder 198.

[0162] In one aspect, at 1922, selecting a bit set of a set of wavelet coefficients to be removed based on an entropy decoding process may also include: determining multiple candidate values ​​for each decoding group in the decoding group based on multiple truncation positions and the MSB of each decoding group, wherein the multiple truncation positions may be based on wavelet frequency bands. For example, selecting a bit set in a set of wavelet coefficients to be removed based on an entropy decoding process at 1714 may also include: determining multiple candidate values ​​for each decoding group in the decoding group based on multiple truncation positions and the MSB of each decoding group, wherein the multiple truncation positions may be based on wavelet frequency bands. The foregoing aspects may correspond to 1018 and / or example 1402. For example, the bit set may correspond to bits in scene 1404 (e.g., scene 0, scene 1, scene 2, etc.). In an example, the MSB may be MSB 1416. In an example, the multiple truncation positions may be or include position 0 1408, position 3 1410, or position 6 1412. In one example, the candidate value may correspond to the scene 1404. In one example, 1922 may be performed by the encoder 198.

[0163] In one aspect, at 1924, selecting a set of bits in a set of wavelet coefficients to be removed based on an entropy decoding process may also include: selecting a candidate value from a plurality of candidate values ​​based on a cutoff position associated with the candidate value and a target bit rate, wherein selecting a set of bits in a set of wavelet coefficients to be removed may be based on the candidate value. For example, selecting a set of bits in a set of wavelet coefficients to be removed based on an entropy decoding process at 1714 may also include: selecting a candidate value from a plurality of candidate values ​​based on a cutoff position associated with the candidate value and a target bit rate, wherein selecting a set of bits in a set of wavelet coefficients to be removed may be based on the candidate value. The foregoing aspects may correspond to 1018, 1020, and 1022. In an example, 1924 may be performed by encoder 198.

[0164] In one aspect, at 1928, serializing the set of wavelet coefficients may include dividing the set of wavelet coefficients into slices. For example, at 1716, serializing the set of wavelet coefficients may include dividing the set of wavelet coefficients into slices. In an example, the slices may be or include the slices 1506. In an example, 1928 may be performed by the encoder 198.

[0165] In one aspect, at 1930, serializing the set of wavelet coefficients may include dividing the slice into substreams, where each substream may correspond to a plurality of lines of the image frame. For example, at 1716, serializing the set of wavelet coefficients may include dividing the slice into substreams, where each substream may correspond to a plurality of lines of the image frame. In an example, the substream may be or include the substream 1508. In an example, 1930 may be performed by the encoder 198.

[0166] In one aspect, at 1932, serializing the set of wavelet coefficients may include removing a set of bits from a substream of a slice of the set of wavelet coefficients. For example, at 1716, serializing the set of wavelet coefficients may include removing a set of bits from a substream of a slice of the set of wavelet coefficients. In an example, the set of (removed) bits may correspond to the scene 1404. In an example, 1932 may be performed by the encoder 198.

[0167] In one aspect, the first device may include at least one of a central processing unit (CPU) or a graphics processor. For example, the CPU may be or include the first CPU 410 and the graphics processor may be or include the first GPU 408.

[0168] In one aspect, at 1936, outputting the bitstream may include sending a bitstream including a set of serialized wavelet coefficients to the second device. For example, at 1718, outputting the bitstream may include sending a bitstream including a set of serialized wavelet coefficients to the second device. In addition, Figure 4 It is shown that the first device 405 can send a bitstream including a set of serialized wavelet coefficients to the second device 407 .

[0169] In one aspect, the first device may be a first wireless device and the second device may be a second wireless device. For example, the first device 405 may be a first wireless device and the second device 407 may be a second wireless device.

[0170] Fig. 20 2000 is a flowchart of an example method of display processing according to one or more techniques of the present disclosure. The method can be performed by an apparatus, such as an apparatus for graphics processing, a GPU, a CPU, a DSP, a second device including a decoder (e.g., a second device 407 including the decoder 406), a wireless communication device, etc. Figure 1-Figure 17 The method may be associated with various advantages at the device, such as facilitating wireless streaming of high-resolution images between two wireless devices in a high-performance, low-latency, and energy-efficient manner. In an example, the method may be performed by the decoder 199.

[0171] At 2002, the second device obtains a bitstream associated with an image frame associated with the first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame. Fig.17At 1720, the decoder 1704 may obtain a bitstream associated with an image frame associated with a first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame. In an example, the first device may be the first device 405. In an example, the image frame may be or include the first frame 418, the second frame 436, the third frame 438, the input image frame 518, and / or the input image 602. In an example, the set of serialized wavelet coefficients may be associated with Fig.15 In an example, the bitstream may correspond to the substream 1508 or the transmission at 428. In an example, the bitstream may correspond to the BG 546. In an example, 2002 may be performed by the decoder 199.

[0172] At 2004, the second device deserializes the set of serialized wavelet coefficients, wherein the set of deserialized wavelet coefficients includes a set of additional bits corresponding to the set of wavelet coefficients associated with the set of serialized wavelet coefficients. Fig.17 At 1722, it is shown that the decoder 1704 can deserialize the set of serialized wavelet coefficients, wherein the set of deserialized wavelet coefficients includes a set of additional bits corresponding to the set of wavelet coefficients associated with the set of serialized wavelet coefficients. In an example, deserializing the set of serialized wavelet coefficients can include aspects described above in conjunction with example 1502. In an example, the set of deserialized wavelet coefficients can be or include reconstructed wavelet coefficients 1514. In an example, the set of additional bits can correspond to bits truncated from the wavelet coefficients 1504 before bitstream generation. In an example, deserializing the set of serialized wavelet coefficients can be or include deserialization 430. In an example, 2004 can be performed by the decoder 199.

[0173] At 2006, the second device reconstructs the image frame based on a set of deserialized wavelet coefficients of an inverse discrete wavelet transform (iDWT) for each of the plurality of horizontal and vertical partitions of the image frame in a single channel. For example, Fig.17 At 1724, the decoder 1704 is shown to reconstruct the image frame in a single pass based on a set of deserialized wavelet coefficients for each of the multiple horizontal and vertical partitions of the image frame, the set of deserialized wavelet coefficients being based on an inverse discrete wavelet transform (iDWT). In an example, the iDWT may include the above combined Fig.16 In an example, the iDWT may be the inverse of the second DWT implementation 902, i.e., the iDWT may generate pixel values ​​based on a set of deserialized wavelet coefficients. In an example, the iDWT may be or include the iDWT 432. In an example, the decoder 199 may perform 2006.

[0174] At 2008, the second device outputs an indication of the reconstructed image frame. For example, Fig.17 The decoder 1704 may output an indication of a reconstructed image frame, shown at 1726. In an example, the reconstructed image frame may correspond to a view of the input 1602 after the iDWT is completed 1606. In an example, 2008 may be performed by the decoder 199.

[0175] Fig.21 2100 is a flowchart of an example method of graphics processing according to one or more techniques of the present disclosure. The method can be performed by an apparatus, such as an apparatus for graphics processing, a GPU, a CPU, a DSP, a second device including a decoder (e.g., a second device 407 including decoder 406), a wireless communication device, etc. Figure 1-Figure 17 The method may be associated with various advantages at the device, for example, facilitating wireless streaming of high-resolution images between two wireless devices in a high-performance, low-latency, and energy-efficient manner. In an example, the method (including the various aspects described in detail below) may be performed by a decoder 199.

[0176] At 2104, the second device obtains a bitstream associated with an image frame associated with the first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame. Fig.17 At 1720, the decoder 1704 may obtain a bitstream associated with an image frame associated with a first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame. In an example, the first device may be the first device 405. In an example, the image frame may be or include the first frame 418, the second frame 436, the third frame 438, the input image frame 518, and / or the input image 602. In an example, the set of serialized wavelet coefficients may be associated with Fig.15 In an example, the bitstream may correspond to the substream 1508 or the transmission at 428. In an example, the bitstream may correspond to the BG 546. In an example, 2104 may be performed by the decoder 199.

[0177] At 2106, the second device deserializes the set of serialized wavelet coefficients, wherein the set of deserialized wavelet coefficients includes a set of additional bits corresponding to the set of wavelet coefficients associated with the set of serialized wavelet coefficients. Fig.17At 1722, it is shown that the decoder 1704 can deserialize the set of serialized wavelet coefficients, wherein the set of deserialized wavelet coefficients includes a set of additional bits corresponding to the set of wavelet coefficients associated with the set of serialized wavelet coefficients. In an example, deserializing the set of serialized wavelet coefficients can include the aspects described above in conjunction with example 1502. In an example, the set of deserialized wavelet coefficients can be or include the reconstructed wavelet coefficients 1514. In an example, the set of additional bits can correspond to bits truncated from the wavelet coefficients 1504 before bitstream generation. In an example, deserializing the set of serialized wavelet coefficients can be or include deserialization 430. In an example, 2106 can be performed by the decoder 199.

[0178] At 2112, the image frame is reconstructed in a single channel for each of a plurality of horizontal and vertical partitions of the image frame based on a set of deserialized wavelet coefficients based on an inverse discrete wavelet transform (iDWT). Fig.17 At 1724, the decoder 1704 is shown to reconstruct the image frame based on a set of deserialized wavelet coefficients based on an inverse discrete wavelet transform (iDWT) for each of the multiple horizontal and vertical partitions of the image frame in a single pass. In an example, the iDWT may include the above combined Fig.16 In an example, the iDWT may be the inverse of the second DWT implementation 902, i.e., the iDWT may generate pixel values ​​based on a set of deserialized wavelet coefficients. In an example, the iDWT may be or include the iDWT 432. In an example, the decoder 199 may perform 2112.

[0179] At 2116, the second device outputs an indication of the reconstructed image frame. For example, Fig.17 The decoder 1704 may output an indication of a reconstructed image frame, shown at 1726. In an example, the reconstructed image frame may correspond to a view of the input 1602 after the iDWT is completed 1606. In an example, 2116 may be performed by the decoder 199.

[0180] In one aspect, reconstructing the image frame may include performing an iDWT on the image frame using one or more 16-bit precision calculations.For example, reconstructing the image frame at 1724 may include performing an iDWT on the image frame using one or more 16-bit precision calculations.

[0181] In one aspect, reconstructing the image frame may include allocating the set of deserialized wavelet coefficients to a 12-10-10 bit storage format based on one or more 16-bit precision calculations. For example, reconstructing the image frame at 1724 may include allocating the set of deserialized wavelet coefficients to a 12-10-10 bit storage format based on one or more 16-bit precision calculations.

[0182] In one aspect, the second device may include at least one of a central processing unit (CPU), a graphics processor, or a digital signal processor (DSP). For example, the CPU may be the second CPU 412, the graphics processor may be the second GPU 416, and the DSP may be the DSP 414.

[0183] In one aspect, at 2108, deserializing the set of serialized wavelet coefficients may include combining substreams of the bitstream into stripes, wherein each substream may correspond to a plurality of lines of an image frame, wherein each stripe may be associated with the set of serialized wavelet coefficients. For example, at 1722, deserializing the set of serialized wavelet coefficients may include combining substreams of the bitstream into stripes, wherein each substream may correspond to a plurality of lines of an image frame, wherein each stripe may be associated with the set of serialized wavelet coefficients. In addition, Fig.15 It is shown that deserializing the set of serialized wavelet coefficients may include combining substreams of the bitstream into stripes, wherein each substream may correspond to a plurality of lines of an image frame, wherein each stripe may be associated with a set of serialized wavelet coefficients. For example, the substream may be substream 1508 and the stripe may include stripe 1512. In an example, 2108 may be performed by decoder 199.

[0184] In one aspect, at 2110, deserializing the set of serialized wavelet coefficients may include adding a set of additional bits to the slice. For example, at 1722, deserializing the set of serialized wavelet coefficients may include adding a set of additional bits to the slice. In an example, 2110 may be performed by the decoder 199.

[0185] In one aspect, the first device may be a first wireless device and the second device may be a second wireless device. For example, the first device 405 may be a first wireless device and the second device 407 may be a second wireless device.

[0186] In one aspect, at 2114, reconstructing the image frame based on the set of deserialized wavelet coefficients may include generating samples in a single channel, wherein the reconstructed image frame may be based on the samples. For example, at 1724, reconstructing the image frame based on the set of deserialized wavelet coefficients may include generating samples in a single channel, wherein the reconstructed image frame may be based on the samples. In an example, the samples may correspond to the inverse of the entropy decoding implementation 1002. In an example, 2114 may be performed by the decoder 199.

[0187] In one aspect, at 2102, obtaining a bitstream associated with an image frame associated with the first device may include wirelessly receiving the bitstream from the first device. For example, obtaining a bitstream associated with an image frame associated with the first device at 1720 may include wirelessly receiving the bitstream from the first device. In an example, 2102 may be performed by a decoder 199.

[0188] In one aspect, at 2118, outputting an indication of the reconstructed image frame may include outputting the reconstructed image frame for display on a display panel. For example, at 1726, outputting an indication of the reconstructed image frame may include outputting the reconstructed image frame for display on a display panel. In an example, the display panel may be or include the display 131. In an example, 2118 may be performed by the decoder 199.

[0189] In one aspect, at 2120, the second device may render a user interface (UI) associated with the image frame based on the reconstructed image frame, wherein the rendering of the UI may utilize an on-chip graphics processor memory. For example, Fig.17 At 1730 , the decoder 1704 is shown to render a user interface (UI) associated with the image frame based on the reconstructed image frame, wherein the rendering of the UI may utilize an on-chip graphics processor memory. In an example, 2120 may be performed by the decoder 199 .

[0190] In various configurations, a method or apparatus for graphics processing is provided. The apparatus may be a GPU, a CPU, or some other processor that can perform graphics processing. In various aspects, the apparatus may be a processing unit 120 within a device 104, or may be some other hardware within the device 104 or another device. The apparatus may include: a unit for generating a set of wavelet coefficients for an image frame based on a discrete wavelet transform (DWT) for each of a plurality of horizontal and vertical partitions of the image frame in a single channel. The apparatus may also include: a unit for selecting a set of bits associated with the set of wavelet coefficients to be removed based on an entropy decoding process. The apparatus may also include a unit for serializing the set of wavelet coefficients, wherein the set of serialized wavelet coefficients does not include a set of bits. The apparatus may also include a unit for outputting a bitstream including the set of serialized wavelet coefficients for a second device. The apparatus may also include a unit for dividing the image frame into a plurality of image slices, wherein generating a set of wavelet coefficients for the image frame includes: performing a plurality of DWTs on each of the plurality of image slices, respectively. The apparatus may also include: a unit for obtaining an image frame before generating the set of wavelet coefficients. The apparatus may further include: a unit for converting the image frame from a red-green-blue (RGB) format to a luminance-chrominance (YUV) format before generating the set of wavelet coefficients and after obtaining the image frame.

[0191] In various configurations, a method or apparatus for graphics processing is provided. The apparatus may be a GPU, a CPU, or some other processor that can perform graphics processing. In various aspects, the apparatus may be a processing unit 120 within a device 104, or may be some other hardware within the device 104 or another device. The apparatus may include a unit for obtaining a bitstream associated with an image frame associated with a first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame. The apparatus may also include a unit for deserializing the set of serialized wavelet coefficients, wherein the set of deserialized wavelet coefficients includes a set of additional bits corresponding to the set of wavelet coefficients associated with the set of serialized wavelet coefficients. The apparatus may also include a unit for reconstructing the image frame based on a set of deserialized wavelet coefficients for each of a plurality of horizontal and vertical partitions of the image frame in a single channel, the set of deserialized wavelet coefficients being based on an inverse discrete wavelet transform (iDWT). The apparatus may also include a unit for outputting an indication of the reconstructed image frame. The apparatus may also include means for rendering a user interface (UI) associated with the image frame based on the reconstructed image frame, wherein the rendering of the UI utilizes an on-chip graphics processor memory.

[0192] It should be understood that the specific order or hierarchy of the blocks / steps in the process, flow chart and / or call flow chart disclosed herein is an illustration of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of the blocks / steps in the process, flow chart and / or call flow chart can be rearranged. In addition, some boxes / steps can be combined and / or omitted. Other blocks / steps can also be added. The attached method claims present the elements of multiple steps in an exemplary order, and are not meant to be limited to the specific order or hierarchy presented.

[0193] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims, wherein, unless otherwise specified, references to elements in the singular are not intended to mean "one and only one", but rather "one or more". The word "exemplary" used herein means "as an example, instance, or illustration". Any aspect described herein as "exemplary" is not necessarily to be construed as being preferred or having advantages over other aspects.

[0194] Unless explicitly stated otherwise, the term "some" refers to one or more, and the term "or" may be interpreted as "and / or" if the context does not dictate otherwise. Combinations such as, "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, for example, the combination of "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, or C," and "A, B, C, or any combination thereof" may be only A, only B, only C, A and B, A and C, B and C, or A, B, and C, wherein any such combination may include one or more members of A, B, or C. All structural and functional equivalents of the elements of the various aspects described are clearly incorporated herein by reference in the full text of this disclosure by those of ordinary skill in the art, and are intended to be covered by the claims. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is clearly stated in the claims. Words such as "module", "mechanism", "element", "device" cannot replace the word "unit". Therefore, no claim element is to be interpreted as a functional unit unless the element is clearly stated using the phrase "unit for...". Unless otherwise specified, the phrase "processor" may refer to "any one of one or more processors" (e.g., one processor in one or more processors, multiple (greater than one) processors in one or more processors, or all of one or more processors) and the phrase "memory" may refer to "any one of one or more memories" (e.g., one memory in one or more memories, several (greater than one) memories in one or more memories, or all memories in one or more memories).

[0195] In one or more examples, the functions described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term "processing unit" is used throughout this disclosure, such a processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any functions, processing units, techniques, or other modules described herein are implemented in software, the functions, processing units, techniques, or other modules described herein may be stored on or transmitted via a computer-readable medium as one or more instructions or codes.

[0196] Computer-readable media may include computer data storage media or communication media, including any media that facilitates the transfer of computer programs from one place to another. In this manner, computer-readable media may generally correspond to: (1) a tangible computer-readable storage medium that is non-transitory; or (2) a communication medium, such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to implement instructions, codes, and / or data structures of the technology described in the present disclosure. As an example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, compact disk read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage, or other magnetic storage devices. As used herein, magnetic disks and optical disks include compact disks (CDs), laser optical disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, wherein magnetic disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media. A computer program product may include a computer-readable medium.

[0197] The technology of the present disclosure can be implemented in various devices or apparatuses, including wireless mobile phones, integrated circuits (ICs) or IC groups (e.g., chipsets). Various components, modules or units are described in the present disclosure to emphasize the functional aspects of devices configured to perform the disclosed technology, but they do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units can be combined in any hardware unit, or provided by a collection of interoperable hardware units (including one or more processors described above) in combination with appropriate software and / or firmware. Accordingly, the term "processor" as used herein may refer to any one of any structure or any other structure that implements the technology described herein. In addition, the technology can be fully implemented in one or more circuits or logic elements.

[0198] The following aspects are merely illustrative and may be combined with other aspects or teachings described herein without limitation.

[0199] Aspect 1 is a method for performing graphics processing at a first device, comprising: generating a set of wavelet coefficients for an image frame based on a discrete wavelet transform (DWT) for each of multiple horizontal and vertical partitions of the image frame in a single channel; selecting a set of bits associated with a set of wavelet coefficients to be removed based on an entropy decoding process; serializing the set of wavelet coefficients, wherein the set of serialized wavelet coefficients does not include a set of bits; and, for a second device, outputting a bit stream including the set of serialized wavelet coefficients.

[0200] Aspect 2 may be combined with aspect 1 and further include: dividing the image frame into a plurality of image slices, wherein generating a set of wavelet coefficients for the image frame includes: performing a plurality of DWTs on each of the plurality of image slices, respectively.

[0201] Aspect 3 can be combined with any one of aspects 1-2, and further comprises: obtaining an image frame before generating the set of wavelet coefficients.

[0202] Aspect 4 can be combined with Aspect 3 and also include: before generating a set of wavelet coefficients and after obtaining the image frame, converting the image frame from a red-green-blue (RGB) format to luminance and chrominance, deserializing the image frame of the set of wavelet coefficients based on an inverse discrete wavelet transform (iDWT); and outputting an indication of the reconstructed image frame.

[0203] Aspect 5 may be combined with any one of aspects 1-4, wherein generating a set of wavelet coefficients comprises generating a set of wavelet coefficients in a single pass without intermediate storage.

[0204] Aspect 6 can be combined with any one of Aspects 1-5, wherein generating a set of wavelet coefficients includes: performing DWT on the image frame based on 16-bit precision calculation to generate a set of wavelet coefficients; and, based on 16-bit precision calculation, allocating the set of wavelet coefficients to a 12-10-10 bit storage format.

[0205] Aspect 7 can be combined with any of Aspects 1-6, wherein selecting a set of bits of a set of wavelet coefficients to be removed based on an entropy decoding process includes: dividing the set of wavelet coefficients into regions, wherein each region of a plurality of regions is associated with a plurality of horizontal lines associated with an image frame; and dividing one of the plurality of regions into a plurality of columns, wherein selecting the set of bits is based on at least one of the plurality of regions or the plurality of columns.

[0206] Aspect 8 can be combined with Aspect 7, wherein selecting a set of bits in a set of wavelet coefficients to be removed based on an entropy decoding process also includes: forming a decoding group for the region, wherein the decoding group for the region includes multiple adjacent wavelet coefficients associated with the same illumination or brightness component of the image frame; converting the decoding group from a signed integer format to a sign and magnitude integer format; and determining the most significant bit (MSB) of the decoding group, wherein selecting the group of bits is based on the MSB of the decoding group.

[0207] Aspect 9 can be combined with Aspect 8, wherein selecting a set of bits of a set of wavelet coefficients to be removed based on an entropy decoding process also includes: determining multiple candidate values ​​for each decoding group in the decoding group based on multiple truncation positions and MSBs, wherein the multiple truncation positions are based on the wavelet frequency band; and selecting a candidate value from the multiple candidate values ​​based on the truncation positions and target bit rate associated with the candidate values, wherein selecting the set of bits of the set of wavelet coefficients to be removed is based on the candidate value.

[0208] Aspect 10 can be combined with any of Aspects 1-9, wherein serializing the set of wavelet coefficients includes: dividing the set of wavelet coefficients into strips; and, dividing the strips into substreams, wherein each substream corresponds to multiple rows of an image frame; and, removing one or more bits from the substream of the strips of the set of wavelet coefficients.

[0209] Aspect 11 may be combined with any one of aspects 1-10, wherein the first device comprises at least one of a central processing unit (CPU) or a graphics processor.

[0210] Aspect 12 may be combined with any of aspects 1-11, wherein outputting the bitstream comprises: sending a bitstream comprising a set of serialized wavelet coefficients to the second device.

[0211] Aspect 13 may be combined with any one of aspects 1-12, wherein the first device is a first wireless device and the second device is a second wireless device.

[0212] Aspect 14 is an apparatus for performing graphics processing at a first device, comprising a memory and a processor coupled to the memory, and based on information stored in the memory, the processor is configured to implement the method in any one of aspects 1-14.

[0213] Aspect 15 can be combined with Aspect 14 and includes that the first device is a first wireless device, the first wireless device includes at least one of a transceiver or an antenna coupled to a processor, and wherein, in order to send the bit stream, the processor is configured to send the bit stream via at least one of the transceiver or the antenna.

[0214] Aspect 16 is a device for graphics processing, comprising a unit for implementing the method as described in any one of aspects 1 to 14.

[0215] Aspect 17 is a computer-readable medium (eg, a non-transitory computer-readable medium) storing a computer-executable code, which, when executed by a processor, causes the processor to implement a method as described in any one of aspects 1-14.

[0216] Aspect 18 is a method for performing graphics processing at a second device, comprising: obtaining a bit stream associated with an image frame associated with a first device, wherein the bit stream includes a set of serialized wavelet coefficients associated with the image frame; deserializing the serialized wavelet coefficient set, wherein the deserialized wavelet coefficient set includes: a set of additional bits corresponding to a set of wavelet coefficients associated with the serialized wavelet coefficient set; reconstructing the image frame based on the deserialized wavelet coefficient set for each of multiple horizontal and vertical partitions of the image frame in a single channel, wherein the deserialized wavelet coefficient set is based on an inverse discrete wavelet transform (iDWT); and outputting an indication of the reconstructed image frame.

[0217] Aspect 19 can be combined with Aspect 18, wherein reconstructing the image frame includes: performing iDWT on the image frame using one or more 16-bit precision calculations; and allocating the deserialized wavelet coefficient set to a 12-10-10 bit storage format based on the one or more 16-bit precision calculations.

[0218] Aspect 20 may be combined with any of aspects 18-19, wherein the second device comprises at least one of a central processing unit (CPU), a graphics processor, or a digital signal processor (DSP).

[0219] Aspect 21 can be combined with any of Aspects 18-20, wherein deserializing a set of serialized wavelet coefficients comprises: combining substreams of the bitstream into strips, wherein each substream corresponds to multiple lines of an image frame, wherein each strip is associated with a set of serialized wavelet coefficients; and, adding a set of additional bits to the strips.

[0220] Aspect 22 may be combined with any one of aspects 18-21, wherein the first device is a first wireless device and the second device is a second wireless device.

[0221] Aspect 23 may be combined with any of aspects 18-22, wherein reconstructing the image frame based on the set of deserialized wavelet coefficients comprises generating samples in a single channel, and wherein the reconstructed image frame is based on the samples.

[0222] Aspect 24 may be combined with any one of aspects 18-23, wherein obtaining a bitstream associated with an image frame associated with the first device comprises: wirelessly receiving the bitstream from the first device.

[0223] Aspect 25 may be combined with any one of aspects 18-24, wherein outputting an indication of the reconstructed image frame comprises outputting the reconstructed image frame for display on a display panel.

[0224] Aspect 26 may be combined with any of Aspects 18-25, and further comprises: rendering a user interface (UI) associated with the image frame based on the reconstructed image frame, wherein the rendering of the UI utilizes an on-chip graphics processor memory.

[0225] Aspect 27 is an apparatus for performing graphics processing at a first device, comprising a memory and a processor coupled to the memory, and based on information stored in the memory, the processor is configured to implement the method in any one of aspects 18-26.

[0226] Aspect 28 can be combined with aspect 27 and include the second device being a second wireless device comprising at least one of a transceiver or an antenna coupled to the processor, and wherein, to obtain the bit stream, the processor is configured to obtain the bit stream via at least one of the transceiver or the antenna.

[0227] Aspect 29 is an apparatus for graphics processing, comprising means for implementing the method as described in any of Aspects 18 to 26.

[0228] Aspect 30 is a computer-readable medium (eg, a non-transitory computer-readable medium) storing a computer-executable code, which, when executed by a processor, causes the processor to implement a method as described in any one of Aspects 18-26.

[0229] Various aspects have been described herein. These and other examples are within the scope of the following claims.

Claims

1. An apparatus for performing graphics processing at a first device, comprising: Memory; as well as a processor coupled to the memory and, based on information stored in the memory, configured to: generating, based on a discrete wavelet transform (DWT), a set of wavelet coefficients for the image frame for each of a plurality of horizontal and vertical partitions of the image frame in a single channel; selecting, based on an entropy decoding process, a set of bits associated with the set of wavelet coefficients to be removed; serializing the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include the set of bits; as well as A bitstream including the set of serialized wavelet coefficients is output to a second device.

2. The device according to claim 1, wherein: The processor is further configured to: The image frame is divided into a plurality of image slices, wherein the set of wavelet coefficients is generated for the image frame, and the processor is configured to perform a plurality of DWTs on each of the plurality of image slices, respectively.

3. The device according to claim 1, wherein: The processor is further configured to: The image frame is obtained prior to generating the set of wavelet coefficients.

4. The device according to claim 3, wherein: The processor is further configured to: Before generating the set of wavelet coefficients and after obtaining the image frame, the image frame is converted from a red-green-blue (RGB) format to a luminance-chrominance (YUV) format.

5. The device according to claim 1, wherein: To generate the set of wavelet coefficients, the processor is configured to generate the set of wavelet coefficients in a single pass without intermediate storage.

6. The device according to claim 1, wherein: To generate the set of wavelet coefficients, the processor is configured to: Performing the DWT on the image frame based on 16-bit precision calculation to generate the set of wavelet coefficients; as well as The set of wavelet coefficients is allocated to a 12-10-10 bit storage format based on the 16-bit precision calculation.

7. The device according to claim 1, wherein: To select the set of bits in the set of wavelet coefficients to be removed based on the entropy decoding process, the processor is configured to: dividing the set of wavelet coefficients into regions, wherein each of the regions is associated with a plurality of horizontal lines associated with the image frame; and Regions of the regions are divided into columns, wherein to select the set of bits, the processor is configured to select the set of bits based on at least one of the regions or the columns.

8. The device according to claim 7, wherein: To select the set of bits in the set of wavelet coefficients to be removed based on the entropy decoding process, the processor is configured to: forming a decoding group for the region, wherein the decoding group for the region includes a plurality of adjacent wavelet coefficients associated with a same illumination or brightness component of the image frame; converting the decoded group from a signed integer format to a sign and magnitude integer format; and A most significant bit (MSB) of the decoding group is determined, wherein to select the set of bits, the processor is configured to select the set of bits based on the MSB of the decoding group.

9. The device according to claim 8, wherein: To select the set of bits in the set of wavelet coefficients to be removed based on the entropy decoding process, the processor is configured to: determining a plurality of candidate values ​​for each of the coding groups based on a plurality of truncation positions and the MSBs of each coding group, wherein the plurality of truncation positions are based on wavelet frequency bands; and A candidate value is selected from the plurality of candidate values ​​based on a truncation position associated with the candidate value and a target bit rate, wherein to select the set of bits in the set of wavelet coefficients to be removed, the processor is configured to select the set of bits based on the candidate value.

10. The device according to claim 1, wherein: To serialize the set of wavelet coefficients, the processor is configured to: Dividing the set of wavelet coefficients into strips; dividing the strip into sub-streams, wherein each sub-stream corresponds to a plurality of lines of the image frame; and The set of bits is removed from the substream of the slice in the set of wavelet coefficients.

11. The device according to claim 1, wherein: The first device includes at least one of a central processing unit (CPU) or a graphics processor.

12. The device according to claim 1, wherein: In order to output the bit stream, the processor is configured to: To the second device, the bitstream including the set of serialized wavelet coefficients is sent.

13. The device according to claim 12, wherein: The first device is a first wireless device comprising at least one of a transceiver or an antenna coupled to the processor, and wherein, to transmit the bit stream, the processor is configured to transmit the bit stream via at least one of the transceiver or the antenna.

14. An apparatus for performing graphics processing at a second device, comprising: Memory; as well as a processor coupled to the memory and, based on information stored in the memory, configured to: obtaining a bitstream associated with an image frame associated with a first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame; deserializing the serialized set of wavelet coefficients, wherein the deserialized set of wavelet coefficients includes a set of additional bits corresponding to a set of wavelet coefficients associated with the serialized set of wavelet coefficients; reconstructing the image frame in a single channel for each of a plurality of horizontal and vertical partitions of the image frame based on the set of deserialized wavelet coefficients, the set of deserialized wavelet coefficients being based on an inverse discrete wavelet transform (iDWT); and An indication of the reconstructed image frame is output.

15. The device according to claim 14, wherein: In order to reconstruct the image frame, the processor is configured to: performing the iDWT on the image frame using one or more 16-bit precision calculations; and Based on the one or more 16-bit precision calculations, the set of deserialized wavelet coefficients is allocated to a 12-10-10 bit storage format.

16. The device according to claim 14, wherein: The second device includes at least one of a central processing unit (CPU), a graphics processor, or a digital signal processor (DSP).

17. The device according to claim 14, wherein: To deserialize the set of serialized wavelet coefficients, the processor is configured to: combining substreams of the bitstream into slices, wherein each substream corresponds to a plurality of lines of the image frame, wherein each slice is associated with a set of serialized wavelet coefficients; and The set of additional bits is added to the slice.

18. The device according to claim 14, wherein: The second device is a second wireless device comprising at least one of a transceiver or an antenna coupled to the processor, and wherein, to obtain the bit stream, the processor is configured to obtain the bit stream via at least one of the transceiver or the antenna.

19. The device according to claim 14, wherein: To reconstruct the image frame based on the set of deserialized wavelet coefficients, the processor is configured to generate samples in the single channel, and wherein, to reconstruct the image frame, the processor is configured to reconstruct the image frame based on the samples.

20. The device according to claim 14, wherein: To output the indication of the reconstructed image frame, the processor is configured to: The reconstructed image frame is output for display on a display panel.

21. The device according to claim 14, wherein: The processor is further configured to: A user interface (UI) associated with the image frame is rendered based on the reconstructed image frame and via an on-chip graphics processor memory.

22. A method for performing graphics processing at a first device, comprising: generating, based on a discrete wavelet transform (DWT), a set of wavelet coefficients for the image frame for each of a plurality of horizontal and vertical partitions of the image frame in a single channel; selecting, based on an entropy decoding process, a set of bits associated with the set of wavelet coefficients to be removed; serializing the set of wavelet coefficients, wherein the serialized set of wavelet coefficients does not include the set of bits; and A bitstream including the set of serialized wavelet coefficients is output to a second device.

23. The method according to claim 22, further comprising: The image frame is divided into a plurality of image slices, wherein generating the set of wavelet coefficients for the image frame comprises: performing a plurality of DWTs on each of the plurality of image slices respectively.

24. The method of claim 22, further comprising: The image frame is obtained prior to generating the set of wavelet coefficients.

25. The method according to claim 24, further comprising: Before generating the set of wavelet coefficients and after obtaining the image frame, the image frame is converted from a red-green-blue (RGB) format to a luminance-chrominance (YUV) format.

26. The method of claim 22, wherein: Generating the set of wavelet coefficients includes generating the set of wavelet coefficients in a single channel without intermediate storage.

27. The method of claim 22, wherein: Generating the set of wavelet coefficients includes: performing the DWT on the image frame based on one or more 16-bit precision calculations to generate the set of wavelet coefficients; and Based on the one or more 16-bit precision calculations, the set of wavelet coefficients is allocated to a 12-10-10 bit storage format.

28. The method of claim 22, wherein: Based on the entropy decoding process, selecting the set of bits to be removed from the set of wavelet coefficients comprises: dividing the set of wavelet coefficients into regions, wherein each of the regions is associated with a plurality of horizontal lines associated with the image frame; and Regions of the regions are divided into columns, wherein selecting the set of bits is based on at least one of the regions or the columns.

29. The method according to claim 28, wherein: Based on the entropy decoding process, selecting the set of bits to be removed from the set of wavelet coefficients further comprises: forming a decoding group for the region, wherein the decoding group for the region includes: a plurality of adjacent wavelet coefficients associated with a same illumination or brightness component of the image frame; converting the decoded group from a signed integer format to a sign and magnitude integer format; and A most significant bit (MSB) of the coding group is determined, wherein selecting the set of bits is based on the MSB of the coding group.

30. A method for performing graphics processing at a second device, comprising: obtaining a bitstream associated with an image frame associated with a first device, wherein the bitstream includes a set of serialized wavelet coefficients associated with the image frame; deserializing the serialized set of wavelet coefficients, wherein the deserialized set of wavelet coefficients includes a set of additional bits corresponding to a set of wavelet coefficients associated with the serialized set of wavelet coefficients; reconstructing the image frame in a single pass for each of a plurality of horizontal and vertical partitions of the image frame based on the set of deserialized wavelet coefficients, the set of deserialized wavelet coefficients being based on an inverse discrete wavelet transform (iDWT); and An indication of the reconstructed image frame is output.

Images