Portrait video relighting on device using ai-generated HDR maps

Abstract

This disclosure provides systems, devices, apparatus, and methods, including computer programs encoded on storage media, for portrait video relighting on device using AI-generated HDR maps. A processor obtains an indication of a panoramic view, and generates, based on the indication, a first HDR image. The processor generates a composited relit image based on image data comprising an image frame and an adjusted light level associated with the first HDR image, and outputs another indication of the composited relit image. A processor obtains a first HDR image at a first bit depth, and generates, based on an inverse quantization function and the first HDR image, a second HDR image at a different, second bit depth. The processor determines an adjusted light level of the second HDR image based on the second HDR image and brightness of an HDR panoramic dataset, and outputs an indication of the adjusted light level.

Description

PORTRAIT VIDEO RELIGHTING ON DEVICE USING AI-GENERATED HDR MAPSCROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of Israel Patent Application No. 309187, entitled “PORTRAIT VIDEO RELIGHTING ON DEVICE USING AI-GENERATED HDR MAPS” and filed on December 7, 2023, which is expressly incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates generally to processing systems, and more particularly, to one or more techniques for graphics processing.INTRODUCTION

[0003] Computing devices often perform graphics and / or display processing (e.g., utilizing a graphics processing unit (GPU), a central processing unit (CPU), a display processor, etc.) to render and display visual content. Such computing devices may include, for example, computer workstations, mobile phones such as smartphones, embedded systems, personal computers, tablet computers, and video game consoles. GPUs are configured to execute a graphics processing pipeline that includes one or more processing stages, which operate together to execute graphics processing commands and output a frame. A central processing unit (CPU) may control the operation of the GPU by issuing one or more graphics processing commands to the GPU. Modern day CPUs are typically capable of executing multiple applications concurrently, each of which may need to utilize the GPU during execution. A display processor may be configured to convert digital information received from a CPU to analog values and may issue commands to a display panel for displaying the visual content. A device that provides content for visual presentation on a display may utilize a CPU, a GPU, and / or a display processor.

[0004] Current techniques for portrait video relighting may not be capable of on-device text to high dynamic range (HDR) 360° image generation. There is a need for improved techniques for portrait video relighting.BRIEF SUMMARY

[0005] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0006] In an aspect of the disclosure, a method, a computer-readable medium, and an apparatus for graphics processing are provided. The apparatus includes a memory; and a processor coupled to the memory and, based on information stored in the memory, the processor is configured to: obtain an indication corresponding to a panoramic view; generate, based on the indication corresponding to the panoramic view, a first high dynamic range (HDR) panoramic image; generate a composited relit image based on image data comprising an image frame and on an adjusted light level associated with the first HDR panoramic image; and output an indication of the composited relit image.

[0007] In another aspect of the disclosure, a method, a computer-readable medium, and an apparatus for graphics processing are provided. The apparatus includes a memory; and a processor coupled to the memory and, based on information stored in the memory, the processor is configured to: obtain a first HDR panoramic image at a first bit depth; generate, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depth that is different from the first bit depth; determine an adjusted light level of the second HDR panoramic image based on the second HDR panoramic image and brightness of an HDR panoramic dataset; and output an indication of the adjusted light level.

[0008] To the accomplishment of the foregoing and related ends, the one or more aspects include the features hereinafter fully described and particularly pointed out in the claims. The following description and the annexed drawings set forth in detail certain illustrative features of the one or more aspects. These features are indicative, however, of but a few of the various ways in which the principles of various aspects may be employed, and this description is intended to include all such aspects and their equivalents.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIG. 1 is a block diagram that illustrates an example content generation system in accordance with one or more techniques of this disclosure.

[0010] FIG. 2 illustrates an example GPU in accordance with one or more techniques of this disclosure.

[0011] FIG. 3 illustrates an example image or surface in accordance with one or more techniques of this disclosure.

[0012] FIG. 4 is a diagram illustrating an example pipeline for text-to-360° panoramic image generation and video portrait relighting in accordance with one or more techniques of this disclosure.

[0013] FIG. 5 is a diagram illustrating example aspects pertaining to text-to-360° panoramic image generation in accordance with one or more techniques of this disclosure.

[0014] FIG. 6 is a diagram illustrating example aspects pertaining to perceptual quantization (PQ) based high dynamic range (HDR) quantization used in text-to-360° panoramic image generation in accordance with one or more techniques of this disclosure.

[0015] FIG. 7 is a diagram illustrating example aspects pertaining to HDR decoding used in text-to-360° panoramic image generation in accordance with one or more techniques of this disclosure.

[0016] FIG. 8 is a diagram illustrating an example of a pre-processing module, a relighting module, a rendering module, and a background composition module in accordance with one or more techniques of this disclosure.

[0017] FIG. 9 is a diagram illustrating example aspects pertaining to text-to-360° panoramic image generation and video portrait relighting in accordance with one or more techniques of this disclosure.

[0018] FIG. 10 is a call flow diagram illustrating example communications between a CPU and a graphics processor in accordance with one or more techniques of this disclosure.

[0019] FIG. 11 is a flowchart of an example method of graphics processing in accordance with one or more techniques of this disclosure.

[0020] FIG. 12 is a flowchart of an example method of graphics processing in accordance with one or more techniques of this disclosure.

[0021] FIG. 13 is a flowchart of an example method of graphics processing in accordance with one or more techniques of this disclosure.

[0022] FIG. 14 is a flowchart of an example method of graphics processing in accordance with one or more techniques of this disclosure.DETAILED DESCRIPTION

[0023] Various aspects of systems, apparatuses, computer program products, and methods are described more fully hereinafter with reference to the accompanying drawings. This disclosure may, however, be embodied in many different forms and should not be construed as limited to any specific structure or function presented throughout this disclosure. Rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art. Based on the teachings herein one skilled in the art should appreciate that the scope of this disclosure is intended to cover any aspect of the systems, apparatuses, computer program products, and methods disclosed herein, whether implemented independently of, or combined with, other aspects of the disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth herein. In addition, the scope of the disclosure is intended to cover such an apparatus or method which is practiced using other structure, functionality, or structure and functionality in addition to or other than the various aspects of the disclosure set forth herein. Any aspect disclosed herein may be embodied by one or more elements of a claim.

[0024] Although various aspects are described herein, many variations and permutations of these aspects fall within the scope of this disclosure. Although some potential benefits and advantages of aspects of this disclosure are mentioned, the scope of this disclosure is not intended to be limited to particular benefits, uses, or objectives. Rather, aspects of this disclosure are intended to be broadly applicable to different wireless technologies, system configurations, processing systems, networks, and transmission protocols, some of which are illustrated by way of example in the figures and in the following description. The detailed description and drawings are merely illustrative of this disclosure rather than limiting, the scope of this disclosure being defined by the appended claims and equivalents thereof.

[0025] Several aspects are presented with reference to various apparatus and methods. These apparatus and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, and the like (collectively referred to as “elements”). These elements may be implemented using electronic hardware, computer software, or anycombination thereof. Whether such elements are implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system.

[0026] By way of example, an element, or any portion of an element, or any combination of elements may be implemented as a “processing system” that includes one or more processors (which may also be referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems-on-chip (SOCs), baseband processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. One or more processors in the processing system may execute software. Software can be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0027] The term application may refer to software. As described herein, one or more techniques may refer to an application (e.g., software) being configured to perform one or more functions. In such examples, the application may be stored in a memory (e.g., on-chip memory of a processor, system memory, or any other memory). Hardware described herein, such as a processor may be configured to execute the application. For example, the application may be described as including code that, when executed by the hardware, causes the hardware to perform one or more techniques described herein. As an example, the hardware may access the code from a memory and execute the code accessed from the memory to perform one or more techniques described herein. In some examples, components are identified in this disclosure. In such examples, the components may be hardware, software, or a combination thereof. The components may be separate components or subcomponents of a single component.

[0028] In one or more examples described herein, the functions described may be implemented in hardware, software, or any combination thereof. If implemented in software, the functions may be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of the aforementioned types of computer-readable media, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.

[0029] As used herein, instances of the term “content” may refer to “graphical content,” an “image,” etc., regardless of whether the terms are used as an adjective, noun, or other parts of speech. In some examples, the term “graphical content,” as used herein, may refer to a content produced by one or more processes of a graphics processing pipeline. In further examples, the term “graphical content,” as used herein, may refer to a content produced by a processing unit configured to perform graphics processing. In still further examples, as used herein, the term “graphical content” may refer to a content produced by a graphics processing unit. As used herein, the terms “panoramic view” or “360° panoramic” may refer to an image with a wide or full view that encompasses directionalities beyond standard portrait or landscape views. As used herein, the term “bit depth” may refer to a number of bits utilized for image representation. As used herein, the terms “quantization function” or “quantization workflow” may refer to a process that converts images from a higher-bit depth to a lower-bit depth through one or more of a PQ transfer function and / or a quantization, while the terms “inverse quantization function” or “quantization workflow” may refer to a process that converts images from a lower-bit depth to a higher-bit depth through one or more of a de-quantization and / or a PQ inverse transfer function. As used herein, the term “light level” may refer to an amount of illumination / brightness present in an image. As used herein, the term “segmentation map” may refer to a visual partition for discrete portions in an image, which represents a specific subject category or a specific area in an image. As used herein, the term “face bounding box” may refer torectangle, defined by a set of four coordinates, inside of which a face of a user / subj ect in an image is contained for image processing operations thereon (e.g., cropping, resizing, rotation, lighting, etc.). As used herein, the term “rotation degrees” may refer to an amount of rotation applied to a generated 360° panoramic image, which is related to the illumination / brightness directionality during video portrait relighting. As used herein, the term “rotation matrix” may refer to a transform comprising a set of values or degrees to transpose or rotate an image in 2D space, or transpose or rotate vectors in 3D space. As used herein, the term “compositing” may refer to combining two or more image elements together into a single image. As used herein, a reference to the term “Al model” may alternately apply to, or may include, a reference to the term “ML model,” or vice versa. As used herein, the term “diffusion network” may refer to a statistical model utilizing latent variables to which observable variables are related. As used herein, the term “latent variable” may refer to a variable that may be inferred or predicted from observable variables.

[0030] A goal of “portrait video relighting” may be to re-illuminate / brighten a subject with consistent lighting to make the subject naturally embedded into a new background. Generative Al may be used for portrait video relighting. Some video portrait video relighting techniques may not be able to perform on-device text to HDR 360° image generation. Some on-device portrait video relighting technology techniques may not be able to leverage generated text to HDR 360° panoramic images.

[0031] Various technologies pertaining to portrait video relighting on a device using artificial intelligence (Al) generated high dynamic range (HDR) maps are described herein. In an example, an apparatus obtains an indication of a panoramic view. The apparatus generates, based on the indication of the panoramic view, a first high dynamic range (HDR) panoramic image at a first bit depth. The apparatus generates, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depth that is different from the first bit depth. The apparatus outputs an indication of an adjusted light level of the second HDR panoramic image based on average brightness of real HDR 360° panoramic datasets (e.g., HDR panoramic datasets, generally). Vis-a-vis generating, based on the indication of the panoramic view, a first high dynamic range (HDR) panoramic image at a first bit depth and generating, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depththat is different from the first bit depth, the apparatus may perform on-device high- quality video portrait relighting. Furthermore, the inverse quantization function may be able to recover a higher amount of HDR luminance compared to other inverse quantization functions.

[0032] The examples describe herein may refer to a use and functionality of a graphics processing unit (GPU). As used herein, a GPU can be any type of graphics processor, and a graphics processor can be any type of processor that is designed or configured to process graphics content. For example, a graphics processor or GPU can be a specialized electronic circuit that is designed for processing graphics content. As an additional example, a graphics processor or GPU can be a general purpose processor that is configured to process graphics content.

[0033] FIG. 1 is a block diagram that illustrates an example content generation system 100 configured to implement one or more techniques of this disclosure. The content generation system 100 includes a device 104. The device 104 may include one or more components or circuits for performing various functions described herein. In some examples, one or more components of the device 104 may be components of a SOC. The device 104 may include one or more components configured to perform one or more techniques of this disclosure. In the example shown, the device 104 may include a processing unit 120, a content encoder / decoder 122, and a system memory 124. In some aspects, the device 104 may include a number of components (e.g., a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131). Display(s) 131 may refer to one or more displays 131. For example, the display 131 may include a single display or multiple displays, which may include a first display and a second display. The first display may be a left-eye display and the second display may be a right-eye display. In some examples, the first display and the second display may receive different frames for presentment thereon. In other examples, the first and second display may receive the same frames for presentment thereon. In further examples, the results of the graphics processing may not be displayed on the device, e.g., the first display and the second display may not receive any frames for presentment thereon. Instead, the frames or graphics processing results may be transferred to another device. In some aspects, this may be referred to as split-rendering.

[0034] The processing unit 120 may include an internal memory 121. The processing unit 120 may be configured to perform graphics processing using a graphics processing pipeline 107. The content encoder / decoder 122 may include an internal memory 123. In some examples, the device 104 may include a processor, which may be configured to perform one or more display processing techniques on one or more frames generated by the processing unit 120 before the frames are displayed by the one or more displays 131. While the processor in the example content generation system 100 is configured as a display processor 127, it should be understood that the display processor 127 is one example of the processor and that other types of processors, controllers, etc., may be used as substitute for the display processor 127. The display processor 127 may be configured to perform display processing. For example, the display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by the processing unit 120. The one or more displays 131 may be configured to display or otherwise present frames processed by the display processor 127. In some examples, the one or more displays 131 may include one or more of a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head-mounted display, or any other type of display device.

[0035] Memory external to the processing unit 120 and the content encoder / decoder 122, such as system memory 124, may be accessible to the processing unit 120 and the content encoder / decoder 122. For example, the processing unit 120 and the content encoder / decoder 122 may be configured to read from and / or write to external memory, such as the system memory 124. The processing unit 120 may be communicatively coupled to the system memory 124 over a bus. In some examples, the processing unit 120 and the content encoder / decoder 122 may be communicatively coupled to the internal memory 121 over the bus or via a different connection.

[0036] The content encoder / decoder 122 may be configured to receive graphical content from any source, such as the system memory 124 and / or the communication interface 126. The system memory 124 may be configured to store received encoded or decoded graphical content. The content encoder / decoder 122 may be configured to receive encoded or decoded graphical content, e.g., from the system memory 124 and / or thecommunication interface 126, in the form of encoded pixel data. The content encoder / decoder 122 may be configured to encode or decode any graphical content.

[0037] The internal memory 121 or the system memory 124 may include one or more volatile or non-volatile memories or storage devices. In some examples, internal memory 121 or the system memory 124 may include RAM, static random access memory (SRAM), dynamic random access memory (DRAM), erasable programmable ROM (EPROM), EEPROM, flash memory, a magnetic data media or an optical storage media, or any other type of memory. The internal memory 121 or the system memory 124 may be a non-transitory storage medium according to some examples. The term “non- transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted to mean that internal memory 121 or the system memory 124 is non-movable or that its contents are static. As one example, the system memory 124 may be removed from the device 104 and moved to another device. As another example, the system memory 124 may not be removable from the device 104.

[0038] The processing unit 120 may be a CPU, a GPU, a GPGPU, or any other processing unit that may be configured to perform graphics processing. In some examples, the processing unit 120 may be integrated into a motherboard of the device 104. In further examples, the processing unit 120 may be present on a graphics card that is installed in a port of the motherboard of the device 104, or may be otherwise incorporated within a peripheral device configured to interoperate with the device 104. The processing unit 120 may include one or more processors, such as one or more microprocessors, GPUs, ASICs, FPGAs, arithmetic logic units (ALUs), DSPs, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuitry, or any combinations thereof. If the techniques are implemented partially in software, the processing unit 120 may store instructions for the software in a suitable, non-transitory computer-readable storage medium, e.g., internal memory 121, and may execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Any of the foregoing, including hardware, software, a combination of hardware and software, etc., may be considered to be one or more processors.

[0039] The content encoder / decoder 122 may be any processing unit configured to perform content decoding. In some examples, the content encoder / decoder 122 may beintegrated into a motherboard of the device 104. The content encoder / decoder 122 may include one or more processors, such as one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuitry, or any combinations thereof. If the techniques are implemented partially in software, the content encoder / decoder 122 may store instructions for the software in a suitable, non-transitory computer-readable storage medium, e.g., internal memory 123, and may execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Any of the foregoing, including hardware, software, a combination of hardware and software, etc., may be considered to be one or more processors.

[0040] In some aspects, the content generation system 100 may include a communication interface 126. The communication interface 126 may include a receiver 128 and a transmitter 130. The receiver 128 may be configured to perform any receiving function described herein with respect to the device 104. Additionally, the receiver 128 may be configured to receive information, e.g., eye or head position information, rendering commands, and / or location information, from another device. The transmitter 130 may be configured to perform any transmitting function described herein with respect to the device 104. For example, the transmitter 130 may be configured to transmit information to another device, which may include a request for content. The receiver 128 and the transmitter 130 may be combined into a transceiver 132. In such examples, the transceiver 132 may be configured to perform any receiving function and / or transmitting function described herein with respect to the device 104.

[0041] Referring again to FIG. 1, in certain aspects, the processing unit 120 may include a portrait generator 198 configured to obtain an indication corresponding to a panoramic view; generate, based on the indication corresponding to the panoramic view, a first high dynamic range (HDR) panoramic image; generate a composited relit image based on image data comprising an image frame and on an adjusted light level associated with the first HDR panoramic image; and output an indication of the composited relit image. The portrait generator 198 may be configured to obtain an indication of a first high dynamic range (HDR) panoramic image at a first bit depth;generate, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depth that is different from the first bit depth; determine an adjusted light level of the second HDR panoramic image based on the second HDR panoramic image and brightness of an HDR panoramic dataset; and output an indication of an the adjusted light level. Although the following description may be focused on graphics processing, the concepts described herein may be applicable to other similar processing techniques.

[0042] A device, such as the device 104, may refer to any device, apparatus, or system configured to perform one or more techniques described herein. For example, a device may be a server, a base station, a user equipment, a client device, a station, an access point, a computer such as a personal computer, a desktop computer, a laptop computer, a tablet computer, a computer workstation, or a mainframe computer, an end product, an apparatus, a phone, a smart phone, a server, a video game platform or console, a handheld device such as a portable video game device or a personal digital assistant (PDA), a wearable computing device such as a smart watch, an augmented reality device, or a virtual reality device, a non-wearable device, a display or display device, a television, a television set-top box, an intermediate network device, a digital media player, a video streaming device, a content streaming device, an in-vehicle computer, any mobile device, any device configured to generate graphical content, or any device configured to perform one or more techniques described herein. Processes herein may be described as performed by a particular component (e.g., a GPU) but in other embodiments, may be performed using other components (e.g., a CPU) consistent with the disclosed embodiments.

[0043] GPUs can process multiple types of data or data packets in a GPU pipeline. For instance, in some aspects, a GPU can process two types of data or data packets, e.g., context register packets and draw call data. A context register packet can be a set of global state information, e.g., information regarding a global register, shading program, or constant data, which can regulate how a graphics context will be processed. For example, context register packets can include information regarding a color format. In some aspects of context register packets, there can be a bit or bits that indicate which workload belongs to a context register. Also, there can be multiple functions or programming running at the same time and / or in parallel. For example,functions or programming can describe a certain operation, e.g., the color mode or color format. Accordingly, a context register can define multiple states of a GPU.

[0044] Context states can be utilized to determine how an individual processing unit functions, e.g., a vertex fetcher (VFD), a vertex shader (VS), a shader processor, or a geometry processor, and / or in what mode the processing unit functions. In order to do so, GPUs can use context registers and programming data. In some aspects, a GPU can generate a workload, e.g., a vertex or pixel workload, in the pipeline based on the context register definition of a mode or state. Certain processing units, e.g., a VFD, can use these states to determine certain functions, e.g., how a vertex is assembled. As these modes or states can change, GPUs may need to change the corresponding context. Additionally, the workload that corresponds to the mode or state may follow the changing mode or state.

[0045] FIG. 2 illustrates an example GPU 200 in accordance with one or more techniques of this disclosure. As shown in FIG. 2, GPU 200 includes command processor (CP) 210, draw call packets 212, VFD 220, VS 222, vertex cache (VPC) 224, triangle setup engine (TSE) 226, rasterizer (RAS) 228, Z process engine (ZPE) 230, pixel interpolator (PI) 232, fragment shader (FS) 234, render backend (RB) 236, L2 cache (UCHE) 238, and system memory 240. Although FIG. 2 displays that GPU 200 includes processing units 220-238, GPU 200 can include a number of additional processing units. Additionally, processing units 220-238 are merely an example and any combination or order of processing units can be used by GPUs according to the present disclosure. GPU 200 also includes command buffer 250, context register packets 260, and context states 261.

[0046] As shown in FIG. 2, a GPU can utilize a CP, e.g., CP 210, or hardware accelerator to parse a command buffer into context register packets, e.g., context register packets 260, and / or draw call data packets, e.g., draw call packets 212. The CP 210 can then send the context register packets 260 or draw call packets 212 through separate paths to the processing units or blocks in the GPU. Further, the command buffer 250 can alternate different states of context registers and draw calls. For example, a command buffer can simultaneously store the following information: context register of context N, draw call(s) of context N, context register of context N+l, and draw call(s) of context N+l.

[0047] GPUs can render images in a variety of different ways. In some instances, GPUs can render an image using direct rendering and / or tiled rendering. In tiled rendering GPUs, an image can be divided or separated into different sections or tiles. After the division of the image, each section or tile can be rendered separately. Tiled rendering GPUs can divide computer graphics images into a grid format, such that each portion of the grid, i.e., a tile, is separately rendered. In some aspects of tiled rendering, during a binning pass, an image can be divided into different bins or tiles. In some aspects, during the binning pass, a visibility stream can be constructed where visible primitives or draw calls can be identified. A rendering pass may be performed after the binning pass. In contrast to tiled rendering, direct rendering does not divide the frame into smaller bins or tiles. Rather, in direct rendering, the entire frame is rendered at a single time (i.e., without a binning pass). Additionally, some types of GPUs can allow for both tiled rendering and direct rendering (e.g., flex rendering).

[0048] In some aspects, GPUs can apply the drawing or rendering process to different bins or tiles. For instance, a GPU can render to one bin, and perform all the draws for the primitives or pixels in the bin. During the process of rendering to a bin, the render targets can be located in GPU internal memory (GMEM). In some instances, after rendering to one bin, the content of the render targets can be moved to a system memory and the GMEM can be freed for rendering the next bin. Additionally, a GPU can render to another bin, and perform the draws for the primitives or pixels in that bin. Therefore, in some aspects, there might be a small number of bins, e.g., four bins, that cover all of the draws in one surface. Further, GPUs can cycle through all of the draws in one bin, but perform the draws for the draw calls that are visible, i.e., draw calls that include visible geometry. In some aspects, a visibility stream can be generated, e.g., in a binning pass, to determine the visibility information of each primitive in an image or scene. For instance, this visibility stream can identify whether a certain primitive is visible or not. In some aspects, this information can be used to remove primitives that are not visible so that the non-visible primitives are not rendered, e.g., in the rendering pass. Also, at least some of the primitives that are identified as visible can be rendered in the rendering pass.

[0049] In some aspects of tiled rendering, there can be multiple processing phases or passes.For instance, the rendering can be performed in two passes, e.g., a binning, a visibility or bin-visibility pass and a rendering or bin-rendering pass. During a visibility pass,a GPU can input a rendering workload, record the positions of the primitives or triangles, and then determine which primitives or triangles fall into which bin or area. In some aspects of a visibility pass, GPUs can also identify or mark the visibility of each primitive or triangle in a visibility stream. During a rendering pass, a GPU can input the visibility stream and process one bin or area at a time. In some aspects, the visibility stream can be analyzed to determine which primitives, or vertices of primitives, are visible or not visible. As such, the primitives, or vertices of primitives, that are visible may be processed. By doing so, GPUs can reduce the unnecessary workload of processing or rendering primitives or triangles that are not visible.

[0050] In some aspects, during a visibility pass, certain types of primitive geometry, e.g., position-only geometry, may be processed. Additionally, depending on the position or location of the primitives or triangles, the primitives may be sorted into different bins or areas. In some instances, sorting primitives or triangles into different bins may be performed by determining visibility information for these primitives or triangles. For example, GPUs may determine or write visibility information of each primitive in each bin or area, e.g., in a system memory. This visibility information can be used to determine or generate a visibility stream. In a rendering pass, the primitives in each bin can be rendered separately. In these instances, the visibility stream can be fetched from memory and used to remove primitives which are not visible for that bin.

[0051] Some aspects of GPUs or GPU architectures can provide a number of different options for rendering, e.g., software rendering and hardware rendering. In software rendering, a driver or CPU can replicate an entire frame geometry by processing each view one time. Additionally, some different states may be changed depending on the view. As such, in software rendering, the software can replicate the entire workload by changing some states that may be utilized to render for each viewpoint in an image. In certain aspects, as GPUs may be submitting the same workload multiple times for each viewpoint in an image, there may be an increased amount of overhead. In hardware rendering, the hardware or GPU may be responsible for replicating or processing the geometry for each viewpoint in an image. Accordingly, the hardware can manage the replication or processing of the primitives or triangles for each viewpoint in an image.

[0052] FIG. 3 illustrates image or surface 300, including multiple primitives divided into multiple bins in accordance with one or more techniques of this disclosure. As shownin FIG. 3, image or surface 300 includes area 302, which includes primitives 321, 322, 323, and 324. The primitives 321, 322, 323, and 324 are divided or placed into different bins, e.g., bins 310, 311, 312, 313, 314, and 315. FIG. 3 illustrates an example of tiled rendering using multiple viewpoints for the primitives 321-324. For instance, primitives 321-324 are in first viewpoint 350 and second viewpoint 351. As such, the GPU processing or rendering the image or surface 300 including area 302 can utilize multiple viewpoints or multi-view rendering.

[0053] As indicated herein, GPUs or graphics processors can use a tiled rendering architecture to reduce power consumption or save memory bandwidth. As further stated above, this rendering method can divide the scene into multiple bins, as well as include a visibility pass that identifies the triangles that are visible in each bin. Thus, in tiled rendering, a full screen can be divided into multiple bins or tiles. The scene can then be rendered multiple times, e.g., one or more times for each bin.

[0054] In aspects of graphics rendering, some graphics applications may render to a single target, i.e., a render target, one or more times. For instance, in graphics rendering, a frame buffer on a system memory may be updated multiple times. The frame buffer can be a portion of memory or random access memory (RAM), e.g., containing a bitmap or storage, to help store display data for a GPU. The frame buffer can also be a memory buffer containing a complete frame of data. Additionally, the frame buffer can be a logic buffer. In some aspects, updating the frame buffer can be performed in bin or tile rendering, where, as discussed above, a surface is divided into multiple bins or tiles and then each bin or tile can be separately rendered. Further, in tiled rendering, the frame buffer can be partitioned into multiple bins or tiles.

[0055] As indicated herein, in some aspects, such as in bin or tiled rendering architecture, frame buffers can have data stored or written to them repeatedly, e.g., when rendering from different types of memory. This can be referred to as resolving and unresolving the frame buffer or system memory. For example, when storing or writing to one frame buffer and then switching to another frame buffer, the data or information on the frame buffer can be resolved from the GMEM at the GPU to the system memory, i.e., memory in the double data rate (DDR) RAM or dynamic RAM (DRAM).

[0056] In some aspects, the system memory can also be system-on-chip (SoC) memory or another chip-based memory to store data or information, e.g., on a device or smart phone, the device 104, a wireless communication device, etc. The system memorycan also be physical data storage that is shared by the CPU and / or the GPU. In some aspects, the system memory can be a DRAM chip, e.g., on a device or smart phone. Accordingly, SoC memory can be a chip-based manner in which to store data.

[0057] In some aspects, the GMEM can be on-chip memory at the GPU, which can be implemented by static RAM (SRAM). Additionally, GMEM can be stored on a device, e.g., a smart phone, the device 104, a wireless communication device, etc. As indicated herein, data or information can be transferred between the system memory or DRAM and the GMEM, e.g., at a device. In some aspects, the system memory or DRAM can be at the CPU or GPU. Additionally, data can be stored at the DDR or DRAM. In some aspects, such as in bin or tiled rendering, a small portion of the memory can be stored at the GPU, e.g., at the GMEM. In some instances, storing data at the GMEM may utilize a larger processing workload and / or consume more power compared to storing data at the frame buffer or system memory.

[0058] FIG. 4 is a diagram 400 illustrating an example pipeline for text-to-360° panoramic image generation and video portrait relighting in accordance with one or more techniques of this disclosure. Aspects described herein may be used for several purposes, such as but without limitation: (1) relighting a high dynamic range (HDR) 360° image generated from a text prompt, (2) rendering pre-defined three dimensional (3D) scenes / objects using an HDR 360° image generated from a text prompt, and / or (3) on-device image / video relighting applications (e.g., improving low- light / unbalanced lighting conditions, providing consistent lighting when applying background replacement during video conferencing, and realistically embedding reconstructed avatars into virtual reality (VR) environments). Aspects of diagram 400 may be performed by a graphics processor, a GPU, and or the like (e.g., the processing unit 120 / the portrait generator 198).

[0059] A goal of “portrait video relighting” may be to re-illuminate / brighten a subject with consistent lighting to make the subject naturally embedded into a new background. Generative Al may be used for portrait video relighting. Some video portrait video relighting techniques may not be able to perform on-device text to HDR 360° image generation (e.g., implemented on a wireless communication device). Some on-device portrait video relighting technology techniques may not be able to leverage generated text to HDR 360° panoramic images. Various aspects described herein pertain to a pipeline that generates an HDR map in real-time based on a text prompt and that usesthe HDR map as a global illumination / brightness input for a portrait video relighting pipeline in order to achieve real-time, realistic, and temporal consistent relit results on a mobile device.

[0060] Some aspects described herein pertain to a real-time portrait video relighting system (which may also be referred to as a video relighting pipeline) using an Al-generated HDR map from a text prompt 412 via a text-to-360° panoramic image generator 402 and a video portrait relighting generator 406. The real-time portrait video relighting system, according to aspects herein, utilizes a novel text-to-360° model 414 and an improved relighting framework.

[0061] The video relighting pipeline described herein may include a first step, e.g., comprising the text-to-360° panoramic image generator 402 and a second step, e.g., comprising the video portrait relighting generator 406. In the first step, text-to-360° panoramic image generation may occur based on the text-to-360° panoramic image generator 402. A pre-trained stable diffusion model, described in further detail herein, may be tuned (e.g., fine-tuned) on HDR images in order to generate a new HDR 360° panoramic image 420 from text prompts, e.g., the text prompt 412 based on a text-to- 360° model 414. To achieve the generation of the HDR 360° panoramic image 420, a perceptual quantization (PQ) HDR quantization workflow may be employed to convert HDR images from a high bit depth to a low bit depth (e.g., 8 bits per channel using an unsigned 8-bit integer (UINT8) format), while preserving wide range information, as quantized HDR 360° images (e.g., an quantized HDR image 416). Furthermore, due to a scarcity of large 360° HDR datasets / HDR panoramic datasets, training may be augmented with non-360° HDR images in order to train a model. Additionally, a variable autoencoder (VAE) in a stable diffusion model may not be configured to work on quantized HDR images, such as the quantized HDR image 416. To address this issue, the VAE may be fine-tuned on quantized HDR images in order to encode and decode (e.g., via an HDR decoder 418 from UINT8 to a 32-bit floating point (FLOAT32) format) HDR images without significant information loss.

[0062] In the second step, e.g., based on the video portrait relighting generator 406, video portrait relighting may be performed using the 360° HDR panoramic image 420 generated in the first step based on the text-to-360° panoramic image generator 402. For instance, the 360° HDR panoramic image 420 that is generated may be used by the video portrait relighting generator 406 to relight a subj ect 423 in a new backgroundenvironment, e.g., as a relit image 408 with a new background. To achieve real-time video relighting speed on devices and stable relighting on live camera frames, inputs 404 may be utilized, including but without limitation, live camera frames 422, a foreground segmentation map 424, a hair / head covering segmentation map 426, a face bounding box, rotation degrees of an HDR map, and / or the like, the efficiency and temporal consistency of a light-adding based relighting framework may be improved.

[0063] FIG. 5 is a diagram 500 illustrating example aspects pertaining to text-to-360° panoramic image generation in accordance with one or more techniques of this disclosure. The diagram 500 may pertain to the first step described above in the context of the text-to-360° panoramic image generator 402 for FIG. 4. Aspects of diagram 500 may be performed by a graphics processor, a GPU, and or the like (e.g., the processing unit 120 / the portrait generator 198).

[0064] In aspects, a text encoder 512 may receive an input for a text prompt 510, as described above for the text-to-360° panoramic image generator 402 in FIG. 4. Encoded text may be provided to a diffusion u-net 514, e.g., a pre-trained stable diffusion architecture for text-to-360° HDR image generation, may be utilized by a latent diffusion model 502. In aspects, a number N of denoising steps, or passes, may be performed for the diffusion u-net 514 to reduce noise. An image decoder 516, e.g., a VAE decoder from the stable diffusion architecture, may be configured to receive the diffusion u-net 514 output and convert a latent representation, e.g., associated with the text prompt 510, to an image. The HDR 360° image may be a denoised latent representation generated by the diffusion u-net that is passed to the image decoder 516 to generate an output quantized HDR 360° image; in aspects, the generated output image may be an HDR 360° image 518.

[0065] An HDR 360° image 520, as a supervision signal 504 for supervising latent stable diffusion training may be utilized, in aspects, and HDR ground-truth images may be quantized using a perceptual quantized (PQ) transfer function 522. The PQ transfer function 522 may be configured to perform PQ-based HDR quantization, in which HDR images, e.g., the HDR image 520, may be converted from a high bit depth to a low bit depth (e.g., 8 bits per channel) while preserving HDR information. The PQ transfer function 522 may be configured to output a quantized HDR 360° image 524 using the PQ-based HDR quantization. An image encoder 526, e.g., a VAE from thestable diffusion architecture, may be configured to convert an image, e.g., the quantized HDR 360° image 524, to a latent representation thereof. At (7) - Diffusion loss, a loss function 528, e.g., for diffusion training, may also be utilized from the stable diffusion architecture. An HDR decoder 506 may be configured to perform HDR decoding. A text-to-360° model quantized HDR prediction may be decoded by the HDR decoder 506 from an unsigned 8-bit integer (UINT8) format to a 32-bit single-precision floating-point (FLOAT32) format. A generated HDR 360° image 508, which may comprise an HDR panorama with radiance values, may be used for relighting, according to aspects herein.

[0066] FIG. 6 is a diagram 600 illustrating example aspects pertaining to perceptual quantization (PQ) based high dynamic range (HDR) quantization used in text-to-360° panoramic image generation in accordance with one or more techniques of this disclosure. Aspects of diagram 600 may be performed by a graphics processor, a GPU, and or the like (e.g., the processing unit 120 / the portrait generator 198).

[0067] In one aspect described herein, an HDR quantization workflow may convert HDR images from a high bit depth to a low bit depth (8 bits per channel) while preserving HDR information. By applying the HDR quantization workflow to HDR 360° datasets, quantized HDR 360° images in UINT8 format may be prepared for training a text-to-360° model. By enabling direct prediction of quantized HDR 360° images in a UINT8 format, a network design of the text-to-360° model may be more computationally efficient.

[0068] In one aspect, as shown for diagram 600, a PQ-based HDR quantization workflow with a defined HDR quantization formula, for a PQ transfer function 604, and a quantizer 606 is described. The PQ-based HDR quantization workflow, as shown by way of example, may be based on the HDR10 standard. The PQ-based HDR quantization workflow may be able to restore a dynamic range of luminance FD up to 200,000 nits for an input image, e.g., an HDR 360° image 602. Equation (I) and equation (II) below describe the PQ transfer function 604 and a defined HDR quantization formula for the quantizer 606, respectively.(Eq. II) D' = I NT [198 x E'],for 8 bit quantizationIn equation (I) above, Fa may be a linear luminance in cd / m2, E may be a non-linear signal value, 7 = 7^ / 10000, mi = 0.1593017578125, m2= 78.84375, ci = 0.8359375, C2 = 18.8515625, and C3 = 18.6875. A resulting output image, e.g., a quantized HDR 360° image 608, may be in a UINT8 format after quantization by the quantizer 606.

[0069] FIG. 7 is a diagram 700 illustrating example aspects pertaining to HDR decoding used in text-to-360° panoramic image generation in accordance with one or more techniques of this disclosure. Aspects of diagram 600 may be performed by a graphics processor, a GPU, and or the like (e.g., the processing unit 120 / the portrait generator 198).

[0070] In one aspect described herein, a prediction made by a text-to-360° model may be decoded from UINT8 to FLOAT32 and then may be used for video portrait relighting. To recover FLOAT32 radiance values, a de-quantization formula of a de-quantizer 704 and a PQ inverse transfer function 706 may be employed for a quantized HDR 360° image 702. A light scaler 708 may be configured to perform light scaling, which may be a linear operation. A scale for the light scaling performed by the light scaler 708 may be calculated based on an average brightness / illumination of a dataset in order to adjust a lighting intensity to be suitable for relighting. Equation (III) and equation (IV) below describe a defined HDR de-quantization formula for the dequantizer 704 and the PQ inverse transfer function 706, respectively.(Eq. Ill) E' = D' / 198, for 8 bit quantiztion(Eq. IV) FD= PQ_EOTF(E") = 10000In equation (IV) above, Fa may be a linear luminance in cd / m2, E may be a non-linear signal value, F = F 10000, mi = 0.1593017578125, m2 = 78.84375, c = 0.8359375, C2 = 18.8515625, and C3 = 18.6875. A resulting output image, e.g., an HDR 360° image 710 for relighting, may be in a FLOAT32 format after light scaling by the light scaler 708.

[0071] With reference to video portrait relighting (e.g., the video portrait relighting generator 406 in FIG. 4), a light-adding based relighting framework may be extended in order to make the light-adding based relighting framework more robust for live cameraframes (e.g., the live camera frames 422 in FIG. 4) and to make the light-adding based relighting framework more efficient to run in real-time on devices (e.g., the device 104, wireless communication devices, etc.).

[0072] In one aspect, temporal consistency of the light-adding based relighting framework may be improved. For instance, an average temporal filter that averages normal predictions of three consecutive frames may be added to the light-adding based relighting framework. A network of the light-adding based relighting framework may be fine-tuned using data augmentation strategies. A fixed-size region of interest (ROI) (e.g., an image captured from a camera) may be used instead of an adaptive ROI (e.g., a face-cropped image).

[0073] In one aspect, efficiency of the light-adding based relighting framework may be improved. For instance, an algorithm used to calculate a low saturation image may be changed. Network quantization may be performed. Light map calculations, Tenderers, and background rendering functions may be converted to OpenGL.

[0074] FIG. 8 is a diagram 800 illustrating an example of a pre-processor 802, a relighting generator 804, a Tenderer 808, and a background composition generator 810 in accordance with one or more techniques of this disclosure. The pre-processor 802, the relighting generator 804, the Tenderer 808, and the background composition generator 810 may be part of or may be included in a light-adding based relighting framework, as described herein. Aspects of diagram 800 may be performed by a graphics processor, a GPU, and or the like (e.g., the processing unit 120 / the portrait generator 198). Diagram 800 may be an aspect of diagram 400 in FIG. 4, diagram 500 in FIG. 5, diagram 600 in FIG. 6, and / or diagram 700 in FIG. 7.

[0075] In aspects, with reference to setting a background for output images that are relit, a generated HDR 360° image 830 (e.g., a one-time generation from a text-to-360° model, as described above), may be provided as an input. A diffuse light integration may be applied to the generated HDR 360° image 830 go generate an integrated diffuse HDR map 832, in some aspects, and a tone mapping may be applied to the generated HDR 360° image 830 go generate an LDR map 834, in some aspects.

[0076] The pre-processor 802 may be configured to receive inputs 812, as described herein, e.g., image data. Inputs 812 may include, but are not limited to, an input image 814 (e.g., a camera live frame), a foreground segmentation map 816, a hair / head covering segmentation map 818, face bounding box, and / or the like. The inputs 812 may beutilized to generate a low saturation image 820, by way of example and not limitation, according to processing of one or more of the inputs 812, e.g., by ‘img x 0.6 + gray_img x 0.4 + 0.05’ or the like, while aspects also provide for other bases of the low saturation image 820, including aspects without calculations for low saturations in which the input image 814 may be used instead. In aspects, an input of HDR rotation degrees may be utilized to generate an HDR rotation matrix 828. Segmentation processing 822 may be performed for the inputs 812 to generate a processed foreground segmentation map 826, and operations 823 (e.g., ROI padding, gamma correction 2.6, masking, resizing, etc.) may be performed for the inputs 812 to generate a masked gamma-corrected foreground image 824.

[0077] The relighting generator 804 may be configured to receive the HDR rotation matrix 828 and the masked gamma-corrected foreground image 824 from the pre-processor 802, and may be configured to receive the integrated diffuse HDR map 832 from the background setup. The masked gamma-corrected foreground image 824 may be provided to a geometry net 838 to generate a normal map 840. The normal map 840 may undergo resizing, ROI cropping, temporal filtering, and / or the like, for input to a diffuse light map calculation 842 along with the HDR rotation matrix 828 and the integrated diffuse HDR map 832. The output of the diffuse light map calculation 842 may be a diffuse light map 844.

[0078] The Tenderer 808 may be configured to receive the input image 814 of the inputs 812 and the low saturation image 820 from the pre-processor 802. The Tenderer 808 may also be configured to receive the diffuse light map 844 from the relighting generator 804 and the processed foreground segmentation map 826 from the pre-processor 802 and to generate, based thereon, a masked diffuse light map 846. The Tenderer 808 may be configured to generate a relit image 852 (e.g., of the subject / user in the input image 814) based on a rendering equation 850. The rendering equation 850 may be associated with relighting an image in linear space. As one example, the input image 814 with gamma correction (e.g., 1 / 2.2), the low saturation image 820 with gamma correction (e.g., 1 / 2.2), the masked diffuse light map 846, and scaling inputs 848 (e.g., a first scaling parameter ‘scalei’ and a second scaling parameter ‘scale?’) may be input for the rendering equation 850. The relit image 852 may be based on application of the first scaling parameter ‘scalei’ to the input image 814 combined with application of the second scaling parameter ‘scale?’ with the low saturation image 820 and themasked diffuse light map 846. In aspects, the relit image 852 may undergo Academy Color Encoding System (ACES) tone mapping and / or gamma correction 2.2.

[0079] The background composition generator 810 may be configured to receive a background rendered version of the generated HDR 360° image 830, based on the LDR map 834, and processed with the HDR rotation matrix 828 to generate a rendered background 854. The background composition generator 810 may be configured to receive the processed foreground segmentation map 826 from the pre-processor 802, and using compositing 856, may be configured to generate a composited relit image with virtual background 858 as an output.

[0080] In one aspect described herein, a PQ-based HDR quantization workflow may be used to prepare quantized HDR 360° images in UINT8 format and to enable an extension of a stable diffusion model from low dynamic range (LDR) to HDR without network modifications. Additionally, a single diffusion network may be employed that may be trained in quantized HDR space. Instant text-to-360° HDR image generation on a device (e.g., such as the device 104, a wireless communication device, etc.) may be achieved without utilizing additional neural networks to extend a low dynamic range. Furthermore, as 360° HDR image datasets may be scarce, a training strategy that leverages non-360° HDR images as a pre-training step followed by fine-tuning on 360° HDR images may be employed.

[0081] In one aspect described herein, on-device high-quality video portrait relighting may be achieved (e.g., such as on the device 104, on a wireless communication device, etc.). Instant generation of text-to-360° HDR images from a user prompt may be combined with relighting, according to aspects.

[0082] As described herein, a PQ quantization with a defined formula may recover HDR luminance up to 200,000 nits, which may facilitate representing real-world luminance and may create more realistic lighting effects, particularly when strong light sources are present. In contrast, the standard quantization formula defined in the HDR10 standard may be capable of recovering HDR luminance up to 25,000 nits, which is less than the 200,000 nits recovered by the defined formula described herein. Equation (V) and Equation (VI) below describe the standard HDR quantization formula defined in the HDR10 standard and the defined HDR quantization formula described for aspects herein.(Eq. V) D' = I NT [219 x E' + 16], for 8 bit quantization(Eq. VI) D' = I NT [198 x E'],for 8 bit quantization

[0083] Some HDR recovery techniques may limit an HDR luminance range during training, and hence may not be able to recover luminance well, particularly when there is sunlight in an outdoor scene.

[0084] A network design of the text-to-360° model may be associated with various additional advantages. For instance, the approach described herein may make use of capabilities of a large diffusion model trained on billions of images for better generalization and quality. Moreover, the approach described herein may generate 360° images with a single diffusion network, whereas other approaches may use two different networks. For instance, a first step may be to generate LDR images from a text prompt via a first network and a second step may be to perform super-resolution and inverse tone mapping via a second network in order to generate high-resolution HDR images. Additionally, using a single network and training the single network in quantized HDR space may be more efficient and may facilitate easy on-device inference on edge devices. Furthermore, the approach described herein may generate images at 512 x 512 resolution, whereas other approaches may generate images at 2048 x 4096 resolution.

[0085] As described above, 360° HDR image datasets may be scarce. Training a model on limited datasets may lead to issues with relighting subjects. To overcome the issue of limited availability of 360° HDR image datasets, aspects herein pertain to leveraging non-360° images as a pre-training step followed by fine-tuning on 360° HDR images. Leveraging non-360° images may help a stable diffusion model to learn light properties across different environments. This two-step approach may increase an amount of HDR data for training and may lead to higher quality 360° image generation for diverse text prompts. Additionally, using a pre-trained VAE network of stable diffusion to encode and decode quantized HDR images may lead to erroneous results. To address this issue, aspects described herein pertain to finetuning a VAE on quantized HDR images to generate high quality images. Furthermore, as described above, perceptual quantization may be performed on both non-360° images and 360° images for training.

[0086] FIG. 9 is a diagram 900 illustrating example aspects pertaining to text-to-360° panoramic image generation and video portrait relighting in accordance with one or more techniques of this disclosure. Aspects of diagram 600 may be performed by a graphics processor, a GPU, and or the like (e.g., the processing unit 120 / the portrait generator 198).

[0087] The portrait video relighting on device using Al-generated HDR maps described herein may be implemented on-device (e.g., on the device 104, on wireless communication devices, etc.) and may achieve high-quality video portrait relighting. Aspects presented herein may also combine instant generation of text-to-360° HDR panoramic images from a user prompt with relighting. Aspects also provide for extensible implementations of the instant generation to other applications and scenarios.

[0088] The framework described herein may produce physically convincing and temporally consistent lighting effects. The framework may allow subjects to be naturally embedded in new Al-generated virtual environments. In aspects, the Al-generated virtual environments may be based on a text prompt 902 provided by a user / subject and generated via a text-to-360° panoramic image generator 904, which may comprise a text-to-360° model 906. In aspects, the text-to-360° model 906 may be Al-based, and may be utilized in association with a pre-trained stable diffusion model in order to generate a new HDR 360° panoramic image 908 from text prompts, e.g., the text prompt 902. To achieve the generation of the HDR 360° panoramic image 908, a PQ HDR quantization workflow may also be utilized, as described herein.

[0089] In aspects, a video portrait relighting generator 910 may be configured to perform video portrait relighting based on the 360° HDR panoramic image 908. As one example, the 360° HDR panoramic image 908 that is generated may be used by the video portrait relighting generator 910 to relight a subject 912 (e.g., from a live camera frame and / or the like) in a new Al-generated background environment such as the 360° HDR panoramic image 908, to generate a relit sequence 914. In aspects, this real-time video relighting may be performed at speed on devices with stable relighting on live camera frames using one or more of foreground segmentation maps, hair / head covering segmentation map 426, face bounding boxes, rotation degrees of an HDR map, and / or the like.

[0090] In comparison to other example portrait relighting implementations, aspects in accordance with one or more techniques of this disclosure provide for additional capabilities. The techniques described herein may be used for various HDR generation and image / video relighting scenarios, such as HDR 360° panoramic map generation given a user prompt, generating an HDR map first and then using the HDR map for relighting and / or rendering, on-device HDR generation, and / or on-device relighting.

[0091] In aspects described herein, an HDR 360° panoramic image may first be generated and may then be used for relighting or may be used in a rendering pipeline. Relit images (e.g., composited relit images) generated by the techniques described herein may be physically convincing while preserving facial details present in input images. Physically convincing properties may be observed by checking if a lighting effect is consistent with a direction of a dominant light source and / or if an overall relight color tone is close to a target HDR environment.

[0092] With respect to portrait relighting, aspects herein provide for image relighting, as well as real-time, on-device (e.g., on the device 104, on a wireless communication device, etc.) video relighting, video consistency, high-quality relighting, target light with 360° HDR maps (e.g., for generative Al and real cases), and HDR rotation degrees for controllability. Yet, other example implementations do not provide for such a full list of capabilities. For instance, other example implementations do not provide for realtime, on-device video relighting or for generative Al-based 360° HDR maps.

[0093] FIG. 10 is a call flow diagram 1000 illustrating example communications between a CPU 1002 and a graphics processor 1004 in accordance with one or more techniques of this disclosure. In aspects, call flow diagram 1000 is described for text-to-360° panoramic image generation and video portrait relighting. In an example, the graphics processor 1004 may be or include a GPU and / or the processing unit 120. In aspects, the CPU 1002 and / or the graphics processor 1004 may comprise a wireless communication device.

[0094] At 1008, the graphics processor 1004 may be configured to obtain an indication corresponding to a panoramic view. In aspects, to obtain the indication, the graphics processor 1004 may be configured to receive an indication 1006 corresponding to the panoramic view from the CPU 1002. In aspects, to obtain the indication corresponding to the panoramic view, the graphics processor 1004 may be configuredto receive text input from a user that describes the panoramic view or receive voice input from the user that describes the panoramic view.

[0095] At 1010, the graphics processor 1004 may be configured to generate, based on the indication 1006 corresponding to the panoramic view, a first HDR panoramic image 1011, e.g., at a first bit depth. In aspects, the first bit depth may correspond to an unsigned 8-bit integer (UINT8). The first HDR panoramic image 1011 may comprise a first 360° HDR image. In aspects, to generate the first HDR panoramic image 1011, the graphics processor 1004 may be configured to predict a set of latent features based on the indication of the panoramic view. In aspects, to generate the first HDR panoramic image 1011, the graphics processor 1004 may be configured to generate the first HDR panoramic image 1011 based on the set of latent features. In aspects, to generate the first HDR panoramic image 1011, the graphics processor 1004 may be configured to obtain an ML model that is based on a quantized set of panoramic HDR images and a quantized set of non-panoramic HDR images, and generate the first HDR panoramic image 1011 based on the ML model and the inverse quantization function.

[0096] After 1010, the graphics processor 1004 may be configured to output an indication of the first HDR panoramic image 1011 based on average brightness of an HDR panoramic dataset(s) (e.g., a real HDR 360° panoramic dataset(s)). In aspects, the graphics processor 1004 may be configured to output the indication of the first HDR panoramic image 1011 by providing the first HDR panoramic image 1011 to the CPU 1002. Accordingly, the CPU 1002 may be configured to obtain / receive an indication of / the first HDR panoramic image 1011 from the graphics processor 1004.

[0097] At 1012, the CPU 1002 may be configured to generate, based on an inverse quantization function and the first HDR panoramic image 1011, a second HDR panoramic image 1017 at a second bit depth that is different from the first bit depth. In aspects, the second bit depth may correspond to a 32-bit floating point number. The second HDR panoramic image 1017 may comprise a second 360° HDR image. In aspects, the first bit depth may be less than the second bit depth.

[0098] At 1013, the CPU 1002 may be configured to determine an adjusted light level of the second HDR panoramic image based on the second HDR panoramic image and brightness of an HDR panoramic dataset. In aspects, the CPU 1002 may be configured to determine the adjusted light level of the second HDR panoramic image via arelighting generator, e.g., as described herein with respect to FIG. 8. At 1014, the CPU 1002 may be configured to output an indication of the adjusted light level, e.g., of the second HDR panoramic image 1017 based on average brightness of an HDR panoramic dataset(s) (e.g., a real HDR 360° panoramic dataset(s)). In aspects, to output the indication of the adjusted light level, the CPU 1002 may be configured to store the indication of the adjusted light level of the second HDR panoramic image 1017 in at least one of the memory, a buffer, or a cache. In aspects, to output the indication of the adjusted light level of the second HDR panoramic image 1017, the CPU 1002 may be configured to transmit / provide the indication of the adjusted light level, e.g., of the second HDR panoramic image 1017.

[0099] At 1016, the CPU 1002 may be configured to adjust a first light level of the second HDR panoramic image 1017 based on the indication of the adjusted light level, e.g., of the second HDR panoramic image 1017. In aspects, to adjust the first light level, the CPU 1002 may be configured to relight the second HDR panoramic image 1017, such as via a relighting generator, e.g., as described herein with respect to FIG. 8. The CPU 1002 may be configured to provide / transmit the second HDR panoramic image 1017 subsequent to relighting. Accordingly, the graphics processor 1004 may be configured to obtain / receive the second HDR panoramic image 1017 from the CPU 1002. In aspects, the graphics processor 1004 may be configured to obtain / receive an indication of / the second HDR panoramic image 1017 at a second bit depth that is different from the first bit depth of the first HDR panoramic image 1011. The second HDR panoramic image 1017 may be based on an inverse quantization function and the first HDR panoramic image 1011, and the second HDR panoramic image 1017 may comprise the adjusted light level. In aspects, 1016 may comprise a portion of 1013 and / or 1014.

[0100] At 1018, the graphics processor 1004 may be configured to generate a composited relit image based on (i) image data comprising an image frame and on (ii) an adjusted light level associated with the first HDR panoramic image. As one example, the graphics processor 1004 may be configured to composite a portion of the image frame with the second HDR panoramic image 1017 based on the image data (e.g., comprising the image frame) to generate the composited relit image. In aspects, the image data may comprise at least one of the image frame that includes a face of a user,a set of segmentation maps, a face bounding box, an indication of an HDR rotation, and / or the like.

[0101] At 1020, the graphics processor may be configured to output an indication of the composited relit image, e.g., for a display. In aspects, to output the indication of the composited relit image for a display, the graphics processor 1004 may be configured to store the indication of the composited relit image in a memory, a buffer, or a cache, and / or may be configured to provide the indication of the composited relit image to the display.

[0102] In aspects, with respect to utilizing, training, and / or tuning / re-tuning Al models and / or ML models, the CPU 1002 and / or the graphics processor 1004 may be configured to obtain a set of panoramic HDR images and a set of non-panoramic HDR images. The set of panoramic HDR images and the set of non-panoramic HDR images may be at the second bit depth. The CPU 1002 and / or the graphics processor 1004 may be configured to quantize, via a quantization function, the set of panoramic HDR images and the set of non-panoramic HDR images. The quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images may be at the first bit depth. In aspects, the first bit depth may be less than the second bit depth. The CPU 1002 and / or the graphics processor 1004 may be configured to generate an ML model based on the quantized set of panoramic HDR images and the quantized set of non- panoramic HDR images. In aspects, to generate the second HDR panoramic image 1017, the CPU 1002 and / or the graphics processor 1004 may be configured to generate the first HDR panoramic image 1011 via the ML model and the inverse quantization function. In aspects, to generate the first HDR panoramic image 1011 may be based on a user input, the ML model, and the inverse quantization function. In aspects, to generate the ML model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images, the CPU 1002 and / or the graphics processor 1004 may be configured to pre-train the ML model based on the quantized set of non-panoramic HDR images. In aspects, to generate the ML model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images, the CPU 1002 and / or the graphics processor 1004 may be configured to tune the ML model based on the quantized set of panoramic HDR images. In some aspects, the ML model may comprise a single diffusion network. The CPU 1002 and / or the graphics processor 1004 may be configured to output anindication of the ML model. In one example, the CPU 1002 may be configured to output / provide the ML model for the graphics processor 1004. In another example, the CPU 1002 may be configured to output / store the ML model in a memory, a cache, a buffer, and / or the like.

[0103] FIG. 11 is a flowchart 1100 of an example method of graphics processing in accordance with one or more techniques of this disclosure. The method may be performed by an apparatus, such as an apparatus for graphics processing, a GPU, a CPU, the device 104, a wireless communication device, and / or the like, as used in connection with the aspects of FIGs. 1-10. The method may be associated with various advantages, such as facilitating on-device high quality video portrait relighting. In an example, the method (including the various aspects detailed below) may be performed by the portrait generator 198.

[0104] At 1102, the apparatus obtains an indication corresponding to a panoramic view. With reference to FIG. 10, at 1008, the graphics processor 1004 may be configured to obtain an indication 1006 (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) corresponding to a panoramic view. In aspects, to obtain the indication 1006 (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9), the graphics processor 1004 may be configured to receive the indication 1006 (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) corresponding to the panoramic view from the CPU 1002. In aspects, to obtain the indication 1006 (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) corresponding to the panoramic view, the graphics processor 1004 may be configured to receive text input (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) from a user that describes the panoramic view or receive voice input (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) from the user that describes the panoramic view. In an example, 1102 may be performed by the portrait generator 198.

[0105] At 1104, the apparatus generates, based on the indication corresponding to the panoramic view, a first HDR panoramic image. With reference to FIG. 10, at 1010, the graphics processor 1004 may be configured to generate, based on the indication 1006 (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) corresponding to the panoramic view, a first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), e.g., at a first bit depth (e.g., UINT8 in FIGs. 4, 6, 7). In aspects, the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) may correspond to an unsigned 8-bit integer (UINT8). The first HDR panoramic image 1011 (e.g., 418in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) may comprise a first 360° HDR image. In aspects, to generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), the graphics processor 1004 may be configured to predict a set of latent features (e.g., via the diffusion u-net 514 in FIG. 5) based on the indication of the panoramic view. In aspects, to generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), the graphics processor 1004 may be configured to generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) based on the set of latent features (e.g., via the diffusion u-net 514 in FIG. 5). In aspects, to generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), the graphics processor 1004 may be configured to obtain an ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) that is based on a quantized set of panoramic HDR images and a quantized set of non-panoramic HDR images, and generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) based on the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) and the inverse quantization function (e.g., 704, 706 in FIG. 7). After 1010, the graphics processor 1004 may be configured to output an indication of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) based on average brightness of an HDR panoramic dataset(s) (e.g., a real HDR 360° panoramic dataset(s)). In aspects, the graphics processor 1004 may be configured to output the indication of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) by providing the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) to the CPU 1002. Accordingly, the CPU 1002 may be configured to obtain / receive an indication of / the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) from the graphics processor 1004. In an example, 1104 may be performed by the portrait generator 198.

[0106] At 1106, the apparatus generates a composited relit image based on image data comprising an image frame and on an adjusted light level associated with the first HDR panoramic image. With reference to FIG. 10, the CPU 1002 may be configured to provide / transmit the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) subsequent to relighting (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). Accordingly, the graphics processor 1004 may be configured to obtain / receive the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) from the CPU 1002. In aspects, the graphics processor 1004 may be configured to obtain / receive an indication of / the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) at a second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) that is different from the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9). The second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may be based on an inverse quantization function (e.g., 704, 706 in FIG. 7) and the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), and the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may comprise the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). At 1018, the graphics processor 1004 may be configured to generate a composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9) based on (i) image data (e.g., 404 in FIG. 4; 812 in FIG. 8) comprising an image frame (e.g., 422 in FIG. 4; 814 in FIG. 8) and on (ii) an adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) associated with the first HDR panoramic image (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9). As one example, the graphics processor 1004 may be configured to composite a portion of the image frame (e.g., 422 in FIG. 4; 814 in FIG. 8)with the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on the image data (e.g., 404 in FIG. 4; 812 in FIG. 8) (e.g., comprising the image frame (e.g., 422 in FIG. 4; 814 in FIG. 8)) to generate the composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9). In aspects, the image data (e.g., 404 in FIG. 4; 812 in FIG. 8) may comprise at least one of the image frame (e.g., 422 in FIG. 4; 814 in FIG. 8) that includes a face of a user, a set of segmentation maps, a face bounding box, an indication of an HDR rotation, and / or the like.. In an example, 1106 may be performed by the portrait generator 198.

[0107] At 1108, the apparatus outputs an indication of the composited relit image. With reference to FIG. 10, at 1020, the graphics processor may be configured to output an indication of the composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9), e.g., for a display. In aspects, to output the indication of the composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9) for a display, the graphics processor 1004 may be configured to store the indication of the composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9) in a memory, a buffer, or a cache, and / or may be configured to provide the indication of the composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9) to the display. In an example, 1108 may be performed by the portrait generator 198.

[0108] In some aspects herein, Al / ML models may be utilized, as described herein. The elements of flowchart 1100 may be performed based on Al / ML models, or may be performed without Al / ML model utilization.

[0109] FIG. 12 is a flowchart 1200 of an example method of graphics processing in accordance with one or more techniques of this disclosure. The method may be performed by an apparatus, such as an apparatus for graphics processing, a GPU, a CPU, the device 104, a wireless communication device, and / or the like, as used in connection with the aspects of FIGs. 1-10. The method may be associated with various advantages, such as facilitating on-device high quality video portrait relighting. In an example, the method (including the various aspects detailed below) may be performed by the portrait generator 198.

[0110] At 1202, the apparatus obtains an indication corresponding to a panoramic view. With reference to FIG. 10, at 1008, the graphics processor 1004 may be configured to obtain an indication 1006 (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) corresponding to a panoramic view. In aspects, to obtain the indication, the graphics processor 1004 may be configured to receive the indication 1006 (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) corresponding to the panoramic view from the CPU 1002. In aspects, to obtain the indication 1006 (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) corresponding to the panoramic view, the graphics processor 1004 may be configured to receive text input (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) from a user that describes the panoramic view or receive voice input (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) from the user that describes the panoramic view. In an example, 1202 may be performed by the portrait generator 198.

[0111] At 1204, the apparatus generates, based on the indication corresponding to the panoramic view, a first HDR panoramic image. With reference to FIG. 10, at 1010, the graphics processor 1004 may be configured to generate, based on the indication 1006 (e.g., 412 in FIG. 4; 510 in FIG. 5; 902 in FIG. 9) corresponding to the panoramic view, a first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), e.g., at a first bit depth (e.g., UINT8 in FIGs. 4, 6, 7). In aspects, the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) may correspond to an unsigned 8-bit integer (UINT8). The first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) may comprise a first 360° HDR image. In aspects, to generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), the graphics processor 1004 may be configured to predict a set of latent features (e.g., via the diffusion u-net 514 in FIG. 5) based on the indication of the panoramic view. In aspects, to generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), the graphics processor 1004 may be configured to generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) based on the set of latent features (e.g., via the diffusion u-net 514 in FIG. 5). In aspects, to generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), the graphics processor 1004 may be configured to obtain an ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) that is based on a quantized set of panoramic HDR images and a quantized set of non-panoramic HDR images, and generate the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) based on the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) and the inverse quantization function (e.g., 704, 706 in FIG. 7). In an example, 1204 may be performed by the portrait generator 198.

[0112] At 1206, the apparatus outputs an indication of the first HDR panoramic image based on average brightness of real HDR 360° panoramic datasets. With reference to FIG. 10, after 1010, the graphics processor 1004 may be configured to output an indication of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) based on average brightness of an HDR panoramic dataset(s) (e.g., a real HDR 360° panoramic dataset(s)). In aspects, the graphics processor 1004 may be configured to output the indication of the first HDRpanoramic image 1011 by providing the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) to the CPU 1002. Accordingly, the CPU 1002 may be configured to obtain / receive an indication of / the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) from the graphics processor 1004. In an example, 1206 may be performed by the portrait generator 198.

[0113] At 1208, the apparatus obtains an indication of a second HDR panoramic image at a second bit depth that is different from the first bit depth of the first HDR panoramic image, where the second HDR panoramic image is based on an inverse quantization function and the first HDR panoramic image, where the second HDR panoramic image comprises an adjusted light level. With reference to FIG. 10, the CPU 1002 may be configured to provide / transmit the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) subsequent to relighting (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). Accordingly, the graphics processor 1004 may be configured to obtain / receive the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) from the CPU 1002. In aspects, the graphics processor 1004 may be configured to obtain / receive an indication of / the second HDR panoramic image 1017 at a second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) that is different from the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9). The second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may be based on an inverse quantization function (e.g., 704, 706 in FIG. 7) and the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), and the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may comprise the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). For instance, at 1012, the CPU 1002 may be configured to generate, based on an inverse quantization function (e.g., 704, 706 in FIG. 7) and the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), a second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) at a second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) that is differentfrom the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7). In aspects, the second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) may correspond to a 32-bit floating point number. The second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may comprise a second 360° HDR image. In aspects, the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) may be less than the second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7). At 1014, the CPU 1002 may be configured to output an indication of an adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on average brightness of real HDR 360° panoramic datasets. In aspects, to output the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9), the CPU 1002 may be configured to store the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) in at least one of the memory, a buffer, or a cache. In aspects, to output the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9), the CPU 1002 may be configured to transmit / provide the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9). At 1016, the CPU 1002 may be configured to adjust a first light level of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9). In aspects, to adjust the first light level of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9), the CPU 1002 may be configured to relight (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9). The CPU 1002 may be configured to provide / transmit the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) subsequent to relighting (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). Accordingly, the graphics processor 1004 may be configured to obtain / receive the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) from the CPU 1002. In an example, 1208 may be performed by the portrait generator 198.

[0114] At 1210, the apparatus generates a composited relit image based on image data comprising an image frame and on an adjusted light level associated with the first HDR panoramic image. In aspects, the apparatus may composite a portion of the image frame with the second HDR panoramic image based on the image data to generate the composited relit image; e.g., the image data may comprise at least one of the image frame that includes a face of a user, a set of segmentation maps, a face bounding box, an indication of an HDR rotation, etc. With reference to FIG. 10, at 1018, the graphics processor 1004 may be configured to generate a composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9) based on (i) image data (e.g., 404 in FIG. 4; 812 in FIG. 8) comprising an image frame (e.g., 422 in FIG. 4; 814 in FIG. 8) and on (ii) an adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) associated with the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9). As one example, the graphics processor 1004 may be configured to composite a portion of the image frame (e.g., 422 in FIG. 4; 814 in FIG. 8)with the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on the image data (e.g., 404 in FIG. 4; 812 in FIG. 8) (e.g., comprising the image frame (e.g., 422 in FIG. 4; 814 in FIG. 8)) to generate the composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9). In aspects, the image data (e.g., 404 in FIG. 4; 812 in FIG. 8) may comprise at least one of the image frame (e.g., 422 in FIG. 4; 814 in FIG. 8)that includes a face of a user, a set of segmentation maps, a face bounding box, an indication of an HDR rotation, and / or the like. In an example, 1210 may be performed by the portrait generator 198.

[0115] At 1212, the apparatus outputs an indication of the composited relit image, e.g., for a display. With reference to FIG. 10, at 1020, the graphics processor may be configured to output an indication of the composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9), e.g., for a display. In aspects, to output the indication of thecomposited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9) for a display, the graphics processor 1004 may be configured to store the indication of the composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9) in a memory, a buffer, or a cache, and / or may be configured to provide the indication of the composited relit image (e.g., 408 in FIG. 4; 858 in FIG. 8; 914 in FIG. 9) to the display. In an example, 1108 may be performed by the portrait generator 198.

[0116] In some aspects herein, Al / ML models may be utilized, as described herein. The elements of flowchart 1200 may be performed based on Al / ML models, or may be performed without Al / ML model utilization.

[0117] FIG. 13 is a flowchart 1300 of an example method of graphics processing in accordance with one or more techniques of this disclosure. The method may be performed by an apparatus, such as an apparatus for graphics processing, a GPU, a CPU, the device 104, a wireless communication device, and / or the like, as used in connection with the aspects of FIGs. 1-10. The method may be associated with various advantages, such as facilitating on-device high quality video portrait relighting. In an example, the method (including the various aspects detailed below) may be performed by the portrait generator 198.

[0118] At 1302, the apparatus obtains a first HDR panoramic image at a first bit depth. With reference to FIG. 10, after 1010, the graphics processor 1004 may be configured to output an indication of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) based on average brightness of an HDR panoramic dataset(s) (e.g., a real HDR 360° panoramic dataset(s)). In aspects, the graphics processor 1004 may be configured to output the indication of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) by providing the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) to the CPU 1002. Accordingly, the CPU 1002 may be configured to obtain / receive an indication of / the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) from the graphics processor 1004. In an example, 1302 may be performed by the portrait generator 198.

[0119] At 1304, the apparatus generates, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depth that is different from the first bit depth. With reference to FIG. 10, at 1012, the CPU1002 may be configured to generate, based on an inverse quantization function (e.g., 704, 706 in FIG. 7) and the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), a second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) at a second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) that is different from the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7). In aspects, the second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) may correspond to a 32-bit floating point number. The second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may comprise a second 360° HDR image. In aspects, the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) may be less than the second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7). In an example, 1304 may be performed by the portrait generator 198.

[0120] At 1306, the apparatus determines an adjusted light level of the second HDR panoramic image based on the second HDR panoramic image and brightness of an HDR panoramic dataset. With reference to FIG. 10, at 1013, the CPU 1002 may be configured to determine an adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on the second HDR panoramic image and brightness of an HDR panoramic dataset. In aspects, the CPU 1002 may be configured to determine the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) via a relighting generator (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), e.g., as described herein with respect to FIG. 8. In an example, 1306 may be performed by the portrait generator 198.

[0121] At 1308, the apparatus outputs an indication of the adjusted light level. With reference to FIG. 10, at 1014, the CPU 1002 may be configured to output an indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), e.g., of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on average brightness of an HDR panoramic dataset(s) (e.g., a real HDR 360° panoramic dataset(s)). In aspects, to output the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), the CPU 1002 may be configured to store theindication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) in at least one of the memory, a buffer, or a cache. In aspects, to output the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9), the CPU 1002 may be configured to transmit / provide the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), e.g., of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9). At 1016, the CPU 1002 may be configured to adjust a first light level of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), e.g., of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9). In aspects, to adjust the first light level, the CPU 1002 may be configured to relight the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9), such as via a relighting generator (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), e.g., as described herein with respect to FIG. 8. The CPU 1002 may be configured to provide / transmit the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) subsequent to relighting (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). Accordingly, the graphics processor 1004 may be configured to obtain / receive the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) from the CPU 1002. In aspects, the graphics processor 1004 may be configured to obtain / receive an indication of / the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) at a second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) that is different from the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9). The second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may be based on an inverse quantization function (e.g., 704, 706 in FIG. 7) and thefirst HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), and the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may comprise the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). In aspects, 1016 may comprise a portion of 1013 and / or 1014. In an example, 1308 may be performed by the portrait generator 198.

[0122] FIG. 14 is a flowchart 1400 of an example method of graphics processing in accordance with one or more techniques of this disclosure. The method may be performed by an apparatus, such as an apparatus for graphics processing, a GPU, a CPU, the device 104, a wireless communication device, and / or the like, as used in connection with the aspects of FIGs. 1-10. The method may be associated with various advantages, such as facilitating on-device high quality video portrait relighting. In an example, the method (including the various aspects detailed below) may be performed by the portrait generator 198.

[0123] At 1402, the apparatus determines if an Al / ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) is to be utilized, trained, tuned, etc. If so, flowchart 1400 may continue to 1404; if not, flowchart 1400 may continue to 1412. In an example, 1402 may be performed by the portrait generator 198.

[0124] At 1404, the apparatus obtains a set of panoramic HDR images and a set of non- panoramic HDR images, where the set of panoramic HDR images and the set of non- panoramic HDR images are at the second bit depth. Referencing aspects in FIG. 10, with respect to utilizing, training, and / or tuning / re-tuning Al models and / or ML models (e.g., 414 in FIG. 4; 502 in FIG. 5), the CPU 1002 and / or the graphics processor 1004 may be configured to obtain a set of panoramic HDR images and a set of non-panoramic HDR images. The set of panoramic HDR images and the set of non- panoramic HDR images may be at the second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7). In an example, 1404 may be performed by the portrait generator 198.

[0125] At 1406, the apparatus quantizes, via a quantization function, the set of panoramic HDR images and the set of non-panoramic HDR images, where the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images are at the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7), where the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) is less than the second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7). Referencing aspects in FIG. 10, the CPU 1002 and / or the graphics processor 1004may be configured to quantize, via a quantization function, the set of panoramic HDR images and the set of non-panoramic HDR images. The quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images may be at the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7). In aspects, the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) may be less than the second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7). In an example, 1406 may be performed by the portrait generator 198.

[0126] At 1408, the apparatus generates a ML model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images. In such aspects, the apparatus may be configured to generate a second HDR panoramic image (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9; 1017 in FIG. 10) (e.g., at 1414) based on generating the first HDR panoramic image (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9; 1011 in FIG. 10) via a user input, the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5), and the inverse quantization function (e.g., 704, 706 in FIG. 7). Referencing aspects in FIG. 10, the graphics processor 1004 may be configured to generate an ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images. In aspects, to generate the second HDR panoramic image, (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9; 1017 in FIG. 10) the graphics processor 1004 may be configured to generate the first HDR panoramic image (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9; 1011 in FIG. 10) via the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) and the inverse quantization function (e.g., 704, 706 in FIG. 7). In some aspects, the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) may comprise a single diffusion network. In an example, 1408 may be performed by the portrait generator 198.

[0127] At 1410, which may comprise the generation at 1208, the apparatus pre-trains and / or tunes the ML model based on the quantized set of non-panoramic HDR images. Referencing aspects in FIG. 10, to generate the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images, the CPU 1002 and / or the graphics processor 1004 may be configured to pre-train the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) based on the quantized set of non-panoramic HDR images. In aspects, to generate the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) based on the quantized set of panoramicHDR images and the quantized set of non-panoramic HDR images, the CPU 1002 and / or the graphics processor 1004 may be configured to tune the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) based on the quantized set of panoramic HDR images. In an example, 1410 may be performed by the portrait generator 198.

[0128] At 1411, the apparatus outputs an indication of ML model. With reference to FIG. 10, the CPU 1002 and / or the graphics processor 1004 may be configured to output an indication of the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5). In one example, the CPU 1002 may be configured to output / provide the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) for the graphics processor 1004. In another example, the CPU 1002 may be configured to output / store the ML model (e.g., 414 in FIG. 4; 502 in FIG. 5) in a memory, a cache, a buffer, and / or the like. In an example, 1411 may be performed by the portrait generator 198.

[0129] At 1412, the apparatus obtains a first HDR panoramic image at a first bit depth. In aspects, the first HDR panoramic image may be based on a user input, the ML model, and the inverse quantization function. With reference to FIG. 10, after 1010, the graphics processor 1004 may be configured to output an indication of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) based on average brightness of an HDR panoramic dataset(s) (e.g., a real HDR 360° panoramic dataset(s)). In aspects, the graphics processor 1004 may be configured to output the indication of the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) by providing the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) to the CPU 1002. Accordingly, the CPU 1002 may be configured to obtain / receive an indication of / the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9) from the graphics processor 1004.. In an example, 1412 may be performed by the portrait generator 198.

[0130] At 1414, the apparatus generates, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depth that is different from the first bit depth. With reference to FIG. 10, at 1012, the CPU 1002 may be configured to generate, based on an inverse quantization function (e.g., 704, 706 in FIG. 7) and the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), a second HDR panoramicimage 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) at a second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) that is different from the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7). In aspects, the second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) may correspond to a 32-bit floating point number. The second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may comprise a second 360° HDR image. In aspects, the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) may be less than the second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7). In an example, 1414 may be performed by the portrait generator 198.

[0131] At 1415, the apparatus determines an adjusted light level of the second HDR panoramic image based on the second HDR panoramic image and brightness of an HDR panoramic dataset. With reference to FIG. 10, at 1013, the CPU 1002 may be configured to determine an adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on the second HDR panoramic image and brightness of an HDR panoramic dataset. In aspects, the CPU 1002 may be configured to determine the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) via a relighting generator (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), e.g., as described herein with respect to FIG. 8. In an example, 1415 may be performed by the portrait generator 198.

[0132] At 1416, the apparatus outputs an indication of an adjusted light level. With reference to FIG. 10, at 1014, the CPU 1002 may be configured to output an indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), e.g., of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on average brightness of an HDR panoramic dataset(s) (e.g., a real HDR 360° panoramic dataset(s)). In aspects, to output the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), the CPU 1002 may be configured to store the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) in at least one of thememory, a buffer, or a cache. In aspects, to output the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9), the CPU 1002 may be configured to transmit / provide the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), e.g., of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9). In an example, 1416 may be performed by the portrait generator 198.

[0133] At 1418, the apparatus adjusts a first light level of the second HDR panoramic image based on the indication of the adjusted light level. With reference to FIG. 10, at 1016, the CPU 1002 may be configured to adjust a first light level, e.g., of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) based on the indication of the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). In aspects, to adjust the first light level of the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9), the CPU 1002 may be configured to relight (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9) the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9), such as via a relighting generator (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9), e.g., as described herein with respect to FIG. 8. In an example, 1418 may be performed by the portrait generator 198.

[0134] At 1420, the apparatus outputs an indication of the second HDR panoramic image subsequent to the adjustment. With reference to FIG. 10, the CPU 1002 may be configured to provide / transmit the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) subsequent to relighting (e.g., 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). Accordingly, the graphics processor 1004 may be configured to obtain / receive the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) from the CPU 1002. In aspects, the graphics processor 1004 may be configured to obtain / receive an indication of / the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) at a second bit depth (e.g., FLOAT32 in FIGs. 4, 6, 7) that is different from the first bit depth (e.g., UINT8 in FIGs. 4, 6, 7) of the first HDR panoramicimage 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9). The second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may be based on an inverse quantization function (e.g., 704, 706 in FIG. 7) and the first HDR panoramic image 1011 (e.g., 418 in FIG. 4; 518 in FIG. 5; 702 in FIG. 7; 830 in FIG. 8; 908 in FIG. 9), and the second HDR panoramic image 1017 (e.g., 420 in FIG. 4; 508 in FIG. 5; 710 in FIG. 7; 854 in FIG. 8; 914 in FIG. 9) may comprise the adjusted light level (e.g., via: 406 in FIG. 4; 708 in FIG. 7; 804 in FIG. 8; 910 in FIG. 9). In an example, 1418 may be performed by the portrait generator 198.

[0135] In some aspects herein, Al / ML models may be utilized, as described above. The following elements of flowchart 1400 may be performed based on Al / ML models, or may be performed without Al / ML model utilization.

[0136] In one aspect, obtaining the indication of the panoramic view may include receiving text input from a user that describes the panoramic view or receiving voice input from the user that describes the panoramic view.

[0137] In one aspect, the first bit depth may correspond to an unsigned 8-bit integer (UINT8), and the second bit depth may correspond to a 32-bit floating point number (FLOAT32).

[0138] In one aspect, the image data may include at least one of an image frame that includes a face of a user, a set of segmentation maps, a face bounding box, or an indication of an HDR rotation.

[0139] In one aspect, the apparatus may composite a portion of the image frame with the second HDR panoramic image based on the set of segmentation maps.

[0140] In one aspect, the apparatus may obtain a set of panoramic HDR images and a set of non-panoramic HDR images, where the set of panoramic HDR images and the set of non-panoramic HDR images may be at the second bit depth.

[0141] In one aspect, the apparatus may quantize, via a quantization function, the set of panoramic HDR images and the set of non-panoramic HDR images, where the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images may be at the first bit depth.

[0142] In one aspect, the apparatus may generate a machine learning (ML) model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images, where generating the second HDR panoramic image may includegenerating the first HDR panoramic image via the ML model and the inverse quantization function.

[0143] In one aspect, generating the ML model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images may include: pretraining the ML model based on the quantized set of non-panoramic HDR images.

[0144] In one aspect, generating the ML model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images may include: tuning the ML model based on the quantized set of panoramic HDR images.

[0145] In one aspect, the ML model may include a single diffusion network.

[0146] In one aspect, the first HDR panoramic image may include a first 360° HDR image, and the second HDR panoramic image may include a second 360° HDR image.

[0147] In one aspect, outputting the indication of the adjusted light level of the second HDR panoramic image may include: storing the indication of the adjusted light level of the second HDR panoramic image in at least one of a memory, a buffer, or a cache; or transmitting the indication of the adjusted light level of the second HDR panoramic image.

[0148] In one aspect, the apparatus may adjust a first light level of the second HDR panoramic image based on the indication of the adjusted light level of the second HDR panoramic image.

[0149] In one aspect, adjusting the first light level of the second HDR panoramic image may include relighting the second HDR panoramic image.

[0150] In one aspect, generating the first HDR panoramic image may include: predicting a set of latent features based on the indication of the panoramic view.

[0151] In one aspect, generating the HDR panoramic image may include: generating the first HDR panoramic image based on the set of latent features.

[0152] In one aspect, the first bit depth may be less than the second bit depth.

[0153] As used herein, the term “latent feature” may refer to a feature embedded within a manifold in which items resembling each other are positioned closer to one another. In an example, a device may receive a text prompt, where the text prompt may be indicative of a panoramic view. The device may process the text prompt. The device may input the processed text prompt and a noisy array of numbers to a machine learning model (e.g., U-Net). The machine learning model may output a set of latentfeatures based on the input. The set of latent features may then be provided to a decoder which may generate an image.

[0154] In configurations, a method or an apparatus for graphics processing is provided. The apparatus may be a GPU, a CPU, or some other processor that may perform graphics processing. In aspects, the apparatus may be the processing unit 120 within the device 104, or may be some other hardware within the device 104 or another device. The apparatus may include means for obtaining an indication of a panoramic view. The apparatus may further include means for generating, based on the indication of the panoramic view, a first high dynamic range (HDR) panoramic image at a first bit depth. The apparatus may further include means for generating, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depth that is different from the first bit depth. The apparatus may further include means for outputting an indication of an adjusted light level of the second HDR panoramic image based on average brightness of real HDR 360° panoramic datasets. The apparatus may further include means for compositing a portion of the image frame with the second HDR panoramic image based on the set of segmentation maps. The apparatus may further include means for obtaining a set of panoramic HDR images and a set of non-panoramic HDR images, where the set of panoramic HDR images and the set of non-panoramic HDR images are at the second bit depth. The apparatus may further include means for quantizing, via a quantization function, the set of panoramic HDR images and the set of non-panoramic HDR images, where the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images are at the first bit depth, where the first bit depth is less than the second bit depth. The apparatus may further include means for generating a machine learning (ML) model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images, where the first HDR panoramic image is based on a user input, the ML model, and the inverse quantization function. The apparatus may further include means for outputting an indication of the ML model. The apparatus may further include means for adjusting a first light level of the second HDR panoramic image based on the indication of the adjusted light level of the second HDR panoramic image. The apparatus may further include means for outputting an indication of the second HDR panoramic image subsequent to the adjustment The apparatus may further include means for outputting an indication ofthe first HDR panoramic image based on average brightness of real HDR 360° panoramic datasets. The apparatus may further include means for obtaining an indication of a second HDR panoramic image at a second bit depth that is different from the first bit depth, where the second HDR panoramic image is based on an inverse quantization function and the first HDR panoramic image, where the second HDR panoramic image comprises an adjusted light level. The apparatus may further include means for compositing a portion of an image frame with the second HDR panoramic image based on image data to generate a composited relit image. The apparatus may further include means for outputting an indication of the composited relit image for a display. The apparatus may further include means for obtaining an indication of a first high dynamic range (HDR) panoramic image at a first bit depth. The apparatus may further include means for generating, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depth that is different from the first bit depth. The apparatus may further include means for outputting an indication of an adjusted light level of the second HDR panoramic image based on average brightness of real HDR 360° panoramic datasets.

[0155] It is understood that the specific order or hierarchy of blocks / steps in the processes, flowcharts, and / or call flow diagrams disclosed herein is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of the blocks / steps in the processes, flowcharts, and / or call flow diagrams may be rearranged. Further, some blocks / steps may be combined and / or omitted. Other blocks / steps may also be added. The accompanying method claims present elements of the various blocks / steps in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0156] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language of the claims, where reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, orillustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0157] Unless specifically stated otherwise, the term “some” refers to one or more and the term “or” may be interpreted as “and / or” where context does not dictate otherwise. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. The words “module,” “mechanism,” “element,” “device,” and the like may not be a substitute for the word “means.” As such, no claim element is to be construed as a means plus function unless the element is expressly recited using the phrase “means for.” Unless stated otherwise, the phrase “a processor” may refer to “any of one or more processors” (e.g., one processor of one or more processors, a number (greater than one) of processors in the one or more processors, or all of the one or more processors) and the phrase “a memory” may refer to “any of one or more memories” (e.g., one memory of one or more memories, a number (greater than one) of memories in the one or more memories, or all of the one or more memories).

[0158] In one or more examples, the functions described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term “processing unit” has been used throughout this disclosure, such processing units may be implemented in hardware, software, firmware, or any combination thereof. If any function, processing unit, technique described herein, or other module is implemented in software, the function, processing unit, technique described herein,or other module may be stored on or transmitted over as one or more instructions or code on a computer-readable medium.

[0159] Computer-readable media may include computer data storage media or communication media including any medium that facilitates transfer of a computer program from one place to another. In this manner, computer-readable media generally may correspond to: (1) tangible computer-readable storage media, which is non-transitory; or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, compact disc-read only memory (CD-ROM), or other optical disk storage, magnetic disk storage, or other magnetic storage devices. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks usually reproduce data magnetically, while discs usually reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. A computer program product may include a computer-readable medium.

[0160] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs, e.g., a chip set. Various components, modules or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily need realization by different hardware units. Rather, as described above, various units may be combined in any hardware unit or provided by a collection of inter-operative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Also, the techniques may be fully implemented in one or more circuits or logic elements.

[0161] The following aspects are illustrative only and may be combined with other aspects or teachings described herein, without limitation.

[0162] Aspect 1 is a method of graphics processing, including: obtaining an indication corresponding to a panoramic view; generating, based on the indication corresponding to the panoramic view, a first high dynamic range (HDR) panoramic image; generating a composited relit image based on image data comprising an image frame and on an adjusted light level associated with the first HDR panoramic image; and outputting an indication of the composited relit image.

[0163] Aspect 2 may be combined with aspect 1, wherein obtaining the indication of the panoramic view includes receiving text input from a user that describes the panoramic view or receiving voice input from the user that describes the panoramic view.

[0164] Aspect 3 may be combined with any of aspects 1-2, further including: obtain an indication of a second HDR panoramic image at a second bit depth that is different from a first bit depth of the first HDR panoramic image, wherein the second HDR panoramic image is based on an inverse quantization function and the first HDR panoramic image, wherein the second HDR panoramic image comprises the adjusted light level.

[0165] Aspect 4 may be combined with aspect 3, wherein generating the first HDR panoramic image includes: obtaining a machine learning (ML) model that is based on a quantized set of panoramic HDR images and a quantized set of non-panoramic HDR images; and generating the first HDR panoramic image based on the ML model and the inverse quantization function.

[0166] Aspect 5 may be combined with any of aspects 1-4, wherein the first bit depth corresponds to an unsigned 8-bit integer (UINT8), and wherein the second bit depth corresponds to a 32-bit floating point number.

[0167] Aspect 6 may be combined with any of aspects 1-5, further including: compositing a portion of an the image frame with the second HDR panoramic image based on the image data to generate a the composited relit image.

[0168] Aspect 7 may be combined with aspect 6, wherein the image data comprises at least one of the image frame that includes a face of a user, a set of segmentation maps, a face bounding box, or an indication of an HDR rotation.

[0169] Aspect 8 may be combined with any of aspects 6-7, wherein the method is performed by a wireless communication device comprising at least one of a transceiver or an antenna coupled to the processor, and wherein obtaining the image data includes obtaining via at least one of the transceiver or the antenna.

[0170] Aspect 9 may be combined with any of aspects 1-8, wherein outputting the indication of the composited relit image includes outputting the indication of the composited relit image for a display.

[0171] Aspect 10 may be combined with any of aspects 1-9, wherein outputting the indication of the composited relit image includes at least one of: storing the indication of the composited relit image in the memory, a buffer, or a cache; or providing the indication of the composited relit image to the display.

[0172] Aspect 11 may be combined with aspect 3, wherein the first HDR panoramic image comprises a first 360° HDR image, and wherein the second HDR panoramic image comprises a second 360° HDR image; or wherein the first bit depth is less than the second bit depth.

[0173] Aspect 12 may be combined with any of aspects 1-11, wherein generating the first HDR panoramic image includes predicting a set of latent features based on the indication of the panoramic view; and generating the first HDR panoramic image based on the set of latent features.

[0174] Aspect 13 is a method of graphics processing, including: obtaining an indication of a first high dynamic range (HDR) panoramic image at a first bit depth; generating, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depth that is different from the first bit depth; determining an adjusted light level of the second HDR panoramic image based on the second HDR panoramic image and brightness of an HDR panoramic dataset; and outputting an indication of an the adjusted light level.

[0175] Aspect 14 may be combined with aspect 13, further including: obtaining a set of panoramic HDR images and a set of non-panoramic HDR images, wherein the set of panoramic HDR images and the set of non-panoramic HDR images are at the second bit depth; quantizing, via a quantization function, the set of panoramic HDR images and the set of non-panoramic HDR images, wherein the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images are at the first bit depth, wherein the first bit depth is less than the second bit depth; generating a machine learning (ML) model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images, wherein the first HDR panoramic image is based on a user input, the ML model, and the inverse quantization function; and outputting an indication of the ML model.

[0176] Aspect 15 may be combined with aspect 14, wherein generating the ML model based on the quantized set of panoramic HDR images and the quantized set of non- panoramic HDR images includes: pre-training the ML model based on the quantized set of non-panoramic HDR images; and tuning the ML model based on the quantized set of panoramic HDR images.

[0177] Aspect 16 may be combined with any of aspects 14-15, wherein the ML model comprises a single diffusion network.

[0178] Aspect 17 may be combined with any of aspects 13-16, wherein outputting the indication of the adjusted light level of the second HDR panoramic image includes at least one of: storing the indication of the adjusted light level of the second HDR panoramic image in at least one of the memory, a buffer, or a cache; or transmitting or providing the indication of the adjusted light level of the second HDR panoramic image.

[0179] Aspect 18 may be combined with any of aspects 13-17, further including: adjusting a first light level of the second HDR panoramic image based on the indication of the adjusted light level of the second HDR panoramic image; and outputting an indication of the second HDR panoramic image subsequent to the adjustment.

[0180] Aspect 19 may be combined with aspect 18, wherein adjusting the first light level of the second HDR panoramic image includes relighting the second HDR panoramic image, wherein the first HDR panoramic image comprises a first 360° HDR image, and wherein the second HDR panoramic image comprises a second 360° HDR image.

[0181] Aspect 20 is an apparatus for graphics processing comprising a memory and a processor coupled to the memory and, based on information stored in the memory, the processor is configured to implement a method as in any of aspects 1-12.

[0182] Aspect 21 may be combined with aspect 1-12 and includes that the apparatus is a wireless communication device comprising at least one of a transceiver or an antenna coupled to the processor, wherein the processor is configured to obtain the image data via at least one of the transceiver or the antenna.

[0183] Aspect 22 is an apparatus for graphics processing including means for implementing a method as in any of aspects 1-12.

[0184] Aspect 23 is a computer-readable medium (e.g., a non-transitory computer-readable medium) storing computer executable code, the computer executable code, whenexecuted by a processor, causes the processor to implement a method as in any of aspects 1-12.

[0185] Aspect 24 is an apparatus for graphics processing comprising a memory and a processor coupled to the memory and, based on information stored in the memory, the processor is configured to implement a method as in any of aspects 14-19.

[0186] Aspect 25 may be combined with aspect 24 and includes that the apparatus is a wireless communication device comprising at least one of a transceiver or an antenna coupled to the processor, wherein the processor is configured to obtain the image data via at least one of the transceiver or the antenna.

[0187] Aspect 26 is an apparatus for graphics processing including means for implementing a method as in any of aspects 14-19.

[0188] Aspect 27 is a computer-readable medium (e.g., a non-transitory computer-readable medium) storing computer executable code, the computer executable code, when executed by a processor, causes the processor to implement a method as in any of aspects 14-19.

[0189] Various aspects have been described herein. These and other aspects are within the scope of the following claims.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. An apparatus for graphics processing, comprising: a memory; and a processor coupled to the memory and, based on information stored in the memory, the processor is configured to: obtain an indication corresponding to a panoramic view; generate, based on the indication corresponding to the panoramic view, a first high dynamic range (HDR) panoramic image; generate a composited relit image based on image data comprising an image frame and on an adjusted light level associated with the first HDR panoramic image; and output an indication of the composited relit image.

2. The apparatus of claim 1, wherein to obtain the indication of the panoramic view, the processor is configured to receive text input from a user that describes the panoramic view or receive voice input from the user that describes the panoramic view.

3. The apparatus of claim 1, wherein the processor is further configured to: obtain an indication of a second HDR panoramic image at a second bit depth that is different from a first bit depth of the first HDR panoramic image, wherein the second HDR panoramic image is based on an inverse quantization function and the first HDR panoramic image, wherein the second HDR panoramic image comprises the adjusted light level.

4. The apparatus of claim 3, wherein to generate the first HDR panoramic image, the processor is configured to: obtain a machine learning (ML) model that is based on a quantized set of panoramic HDR images and a quantized set of non-panoramic HDR images; and generate the first HDR panoramic image based on the ML model and the inverse quantization function.

5. The apparatus of claim 3, wherein the first bit depth corresponds to an unsigned 8-bit integer (UINT8), and wherein the second bit depth corresponds to a 32-bit floating point number.

6. The apparatus of claim 3, wherein the processor is further configured to: composite a portion of the image frame with the second HDR panoramic image based on the image data to generate the composited relit image.

7. The apparatus of claim 6, wherein the image data comprises at least one of the image frame that includes a face of a user, a set of segmentation maps, a face bounding box, or an indication of an HDR rotation.

8. The apparatus of claim 6, wherein the apparatus is a wireless communication device comprising at least one of a transceiver or an antenna coupled to the processor, and wherein the processor is further configured to obtain the image data via at least one of the transceiver or the antenna.

9. The apparatus of claim 6, wherein to output the indication of the composited relit image, the processor is further to: output the indication of the composited relit image for a display.

10. The apparatus of claim 9, wherein to output the indication of the composited relit image, the processor is configured to: store the indication of the composited relit image in the memory, a buffer, or a cache; or provide the indication of the composited relit image to the display.

11. The apparatus of claim 3, wherein the first HDR panoramic image comprises a first 360° HDR image, and wherein the second HDR panoramic image comprises a second 360° HDR image; or wherein the first bit depth is less than the second bit depth.

12. The apparatus of claim 1, wherein to generate the first HDR panoramic image, the processor is configured to: predict a set of latent features based on the indication of the panoramic view; and generate the first HDR panoramic image based on the set of latent features.

13. An apparatus for graphics processing, comprising: a memory; and a processor coupled to the memory and, based on information stored in the memory, the processor is configured to: obtain a first high dynamic range (HDR) panoramic image at a first bit depth; generate, based on an inverse quantization function and the first HDR panoramic image, a second HDR panoramic image at a second bit depth that is different from the first bit depth; determine an adjusted light level of the second HDR panoramic image based on the second HDR panoramic image and brightness of an HDR panoramic dataset; and output an indication of the adjusted light level.

14. The apparatus of claim 13, wherein the processor is further configured to: obtain a set of panoramic HDR images and a set of non-panoramic HDR images, wherein the set of panoramic HDR images and the set of non-panoramic HDR images are at the second bit depth; quantize, via a quantization function, the set of panoramic HDR images and the set of non-panoramic HDR images, wherein the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images are at the first bit depth, wherein the first bit depth is less than the second bit depth; generate a machine learning (ML) model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images, wherein the first HDR panoramic image is based on a user input, the ML model, and the inverse quantization function; and output an indication of the ML model.

15. The apparatus of claim 14, wherein to generate the ML model based on the quantized set of panoramic HDR images and the quantized set of non-panoramic HDR images, the processor is configured to: pre-train the ML model based on the quantized set of non-panoramic HDR images; and tune the ML model based on the quantized set of panoramic HDR images.

16. The apparatus of claim 14, wherein the ML model comprises a single diffusion network.

17. The apparatus of claim 13, wherein to output the indication of the adjusted light level of the second HDR panoramic image, the processor is configured to: store the indication of the adjusted light level of the second HDR panoramic image in at least one of the memory, a buffer, or a cache; or transmit the indication of the adjusted light level of the second HDR panoramic image.

18. The apparatus of claim 13, wherein the processor is further configured to: adjust a first light level of the second HDR panoramic image based on the indication of the adjusted light level; and output an indication of the second HDR panoramic image subsequent to the adjustment.

19. The apparatus of claim 18, wherein to adjust the first light level of the second HDR panoramic image, the processor is configured to relight the second HDR panoramic image, wherein the first HDR panoramic image comprises a first 360° HDR image, and wherein the second HDR panoramic image comprises a second 360° HDR image.

20. A method of graphics processing, comprising: obtaining an indication corresponding to a panoramic view; and generating, based on the indication corresponding to the panoramic view, a first high dynamic range (HDR) panoramic image;generating a composited relit image based on image data comprising an image frame and on an adjusted light level associated with the first HDR panoramic image; and outputting an indication of the composited relit image.

Images