Surface lightfield compression

Abstract

A light field data set with surface layers is received. Each surface layer includes depth, alpha and spherical harmonics (SH) coefficient data. The depth data, alpha map data, and SH coefficient data are converted to depth, alpha and multiple SH coefficient images. Encoder-side operations are performed on the multiple SH coefficient images to generate an alternative SH coefficient data representation different from an input SH coefficient data representation. The alternative SH coefficient data representation is encoded into a coded bitstream along with the depth and alpha images.

Description

SURFACE LIGHTFIELD COMPRESSION CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Application No.63 / 559,376, filed on February 29, 2024 and European Application No.24176113.9 filed on May 15, 2024, both of which are incorporated by reference herein in their entirety. TECHNOLOGY

[0002] The present invention relates generally to lightfield coding and more particularly to surface lightfield compression and decompression operations. BACKGROUND OF THE INVENTION

[0003] Video techniques are being developed to support transmitting and rendering three dimensional (3D) video content based on available bandwidths supported by contemporary computing and network infrastructure. For example, MPEG video encoders and decoders may be extended or reused to support encoding and decoding MPEG based 3D video content for rendering with a wide variety of computing devices incorporating MPEG codecs. Other video encoders and decoders may also be implemented or developed to support encoding and decoding non-MPEG based 3D video content for rendering with computing devices incorporating non-MPEG codecs.

[0004] A consumer device such as a handheld or wearable device may be installed or configured with video codecs of relatively limited capabilities. If 3D video content is not encoded and delivered in an expected video format, the device will likely be incapable of finding a suitable video decoder to decode the 3D video content. Even if rendered, the decoded 3D video content may include incorrect or inaccurate interpretation or representation of the original 3D video content, and may produce visible artifacts in shapes, colors and luminance values in rendered images.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not assume to have beenrecognized in any prior art on the basis of this section, unless otherwise indicated. BRIEF DESCRIPTION OF DRAWINGS

[0006] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0007] FIG.1A illustrates an example layered surface lightfield (LSLF) representation or set; FIG.1B illustrates example LSLF layers including occluded points; FIG.1C illustrates an example coordinates system (Z-up orientation) for capturing or representing direction- or view-dependent visual data; FIG.1D illustrates an example rendering of an LSLF set; FIG. 1E through FIG.1G illustrate example depth data, alpha data and visual data, respectively, in an LSLF set with 3 layers; FIG.1H illustrates example spherical harmonics (SH) coefficients for a layer of an LSLF set; FIG.1I illustrates an example packed image into which SH coefficients are packed;

[0008] FIG.2A illustrates an end-to-end LSLF compression workflow; FIG.2B illustrates example point cloud compression by separating geometry and attribute compression; FIG.2C illustrates example geometry based point cloud compression (G-PCC) operations;

[0009] FIG.3A illustrates example coding blocks (CBs) and coding tree blocks (CTBs) for coding images packed with depth, alpha and SH coefficients; FIG.3B illustrates an example division of a CB into coefficient groups and sub-blocks; FIG.3C illustrates example scanning orders for scanning coefficient groups or sub-blocks in a CB;

[0010] FIG.4A and FIG.4B illustrate example process flows; and

[0011] FIG.5 illustrates an example hardware platform on which a computer or a computing device as described herein may be implemented.DETAILED DESCRIPTION OF THE INVENTION

[0012] Example embodiments, which relate to surface lightfield compression and decompression operations, are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating the present invention.

[0013] Example embodiments are described herein according to the following outline: 1. GENERAL OVERVIEW 2. LAYERED SURFACE LIGHTFIELDS 3. DEPTH, TRANSPARENCY AND TEXTURE INFORMATION 4. BASIS FUNCTIONS AND COEFFICIENTS 5. LSLF COMPRESSION OPERATIONS 6. HIGHER ORDER COEFFICIENT PACKING 7. CODING BLOCK BASED COMPRESSION 8. POINT CLOUD COMPRESSION 9. NEURAL NETWORK BASED GEOMETRY CODING 10. MESH CODING 11. FLOATING POINT PLANE BASED COMPRESSION 12. TRANSFORM DOMAIN 3D SH CUBE 13. SPARSE MATRIX, HDF5 AND SPATIAL FREQUENCY 14. COEFFICIENTS REGENERATION 15. EXAMPLE PROCESS FLOWS 16. IMPLEMENTATION MECHANISMS – HARDWARE OVERVIEW 17. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS 1. GENERAL OVERVIEW

[0014] This overview presents a basic description of some aspects of an example embodiment of the present invention. It should be noted that this overview is not an extensive or exhaustive summary of aspects of the example embodiment. Moreover, it should be noted that this overview is not intended to be understood as identifying anyparticularly significant aspects or elements of the example embodiment, nor as delineating any scope of the example embodiment in particular, nor the invention in general. This overview merely presents some concepts that relate to the example embodiment in a condensed and simplified format, and should be understood as merely a conceptual prelude to a more detailed description of example embodiments that follows below.

[0015] Augmented reality (AR), virtual reality (VR), mixed reality (MR), and other immersive volumetric video applications have demonstrated many types of unique and outstanding viewing experience. These viewing experiences may be delivered through or enabled by a variety of different types of technologies. A layered surface light field (LSLF) data set as described herein can be used to represent a 3D (visual) scene. The LSLF set includes surface layers with their respective 3D position or geometry data as well as (e.g., visual, etc.) attributes, such as intensity, color, reflectance, etc.

[0016] The attributes in the LSLF set may be direction dependent and may be decomposed into a representation format using a series of spherical harmonics (SH) basis functions with respective SH coefficients. The series of SH basis functions may be reduced or approximated by a subset of all SH basis functions from the lowest order (or 0th order) up to a specific maximum (SH basis function) order. For the purpose of illustration only, as an example, SH basis functions and coefficients may be used to represent or capture view- or direction-dependent light information. It should be noted that in other operational scenarios, non-SH basis functions may be used as represent or capture view- or direction-dependent light information instead of or in addition to SH basis functions.

[0017] Techniques as described herein may be implemented to compress and decompress depth data, alpha map data and SH coefficients derived from an input or original lightfield (e.g., an input or original LSLF set, etc.) relatively efficiently. Some or all of these techniques are specifically optimized for processing higher (e.g., than the 0th, etc.) order SH coefficients in the input LSLF set.

[0018] For example, compressed depth data, alpha map data and SH coefficient data may be generated with one or more of the approaches described herein and encoded as compressed LSLF data in a coded bitstream. Additionally, optionally or alternatively, optimized values for operational parameters can be generated as a part of the LSLF data processing or compression operations and encoded as LSLF metadata in the coded bitstream.

[0019] The LSLF data and metadata may be decompressed or decoded from the coded bitstream to enable downstream recipient devices to generate reconstructed depth data, alpha map data and SH coefficients in a reconstructed LSLF set approximating the input or originalLSLF set. Display image(s) may be generated by the downstream devices from the reconstructed LSLF set for rendering on image displays operating with the downstream devices.

[0020] LSLF data processing or compression operations as described herein may be implemented or performed with a wide variety of new or existing (e.g., enhanced, deployed etc.) image or video codecs including but not limited to those codecs optimized to process two dimensional (2D) images or videos.

[0021] Example embodiments described herein relate to encoding 3D visual content. An input light field data set is received. The input light field data set includes one or more surface layers. Each surface layer in the one or more surface layers of the input light field data set includes depth data, alpha map data, multiple sets of spherical harmonics (SH) coefficients of multiple different combinations of SH basis function order and SH basis function index, etc. The depth data, alpha map data, and the multiple sets of SH coefficients (or non-SH coefficients for selected non-SH basis functions in other operational scenarios) to a depth image, are converted into an alpha map image, and multiple SH coefficient images. One or more encoder-side operations are performed on at least one of the multiple SH coefficient images to generate an alternative SH coefficient data representation different from an input SH coefficient data representation in which the at least one of the multiple SH coefficient images is represented. The alternative SH coefficient data representation is encoded, in place of the at least one of the multiple SH coefficient images, into a coded bitstream along with the depth image and the alpha image. The coded bitstream causes a recipient device of the coded bitstream to generate a reconstructed light field data set that approximates the input light field data set.

[0022] Example embodiments described herein relate to decoding 3D visual content. An alternative SH coefficient data representation, different from an original SH coefficient data representation used to derive the alternative SH coefficient data representation, is decoded from a coded bitstream along with a depth image and an alpha image. The alterative SH coefficient data representation is generated by an upstream device in place of at least one of multiple SH coefficient images in an input light field data set. Based at least in part on the depth image, the alpha map image, and the alternative SH coefficient data representation, a reconstructed light field data set that approximates the input light field data set is generated. The reconstructed light field data set is used to generate a display image from the reconstructed light field data set. The display image is rendered on an image display.

[0023] In some example embodiments, mechanisms as described herein form a part of amedia processing system, including but not limited to any of: a wearable device, a handheld computing device, game machine, television, laptop computer, netbook computer, tablet computer, desktop computer, computer workstation, or various other kinds of computing devices and media processing units.

[0024] Various modifications to the preferred embodiments and the generic principles and features described herein will be readily apparent to those skilled in the art. Thus, the disclosure is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features described herein. 2. LAYERED SURFACE LIGHTFIELDS

[0025] Layered surface lightfield (LSLF), which provides a layered representation of a scene, may be used to target, support or perform relatively high quality novel view synthesis (NVS).

[0026] Multiple cameras – e.g., cameras “-3”, “-2”, … “2”, “3” as illustrated in FIG.1A, etc. – with overlapping fields-of-view from different spatial positions and / or different spatial directions / orientations may be used as a set to capture different images in a set of different camera views. These images of different camera views can be used to generate a volumetric representation of the scene such as an LSLF representation. This LSLF based volumetric representation provides a relatively compact way of representing the volume associated with the scene. Each LSLF representation can be built for a specific target view, for example around a (e.g., geometric, etc.) center of the set of camera views.

[0027] FIG.1A illustrates an example LSLF representation (or a corresponding data set) depicting a scene with multiple surface layers (or corresponding surface lightfields) such as surface layers 1 through 4. Each surface layer among the multiple surface layers includes a collection of (spatial) points forming a specific surface geometry in the volume of the scene with implied connectivity – e.g., all these points of the same surface layer with implied connectivity may be belong to a single continuous or contiguous surface even when the single contiguous surface may have spatial (gap) portions – e.g., in surface layer 4 of FIG.1A – not visible in any camera view in a set of different camera views used to construct the multiple surface layers of lightfields in the LSLF representation. In some operational scenarios, no points in these spatial (gap) portions not visible in any camera view in the set of different camera views are included or represented in the surface layer. Example LSLF representations and operations are described in U.S. Provisional Patent Application No. __ / ___,___ (Attorney Docket No.60175-0557; D23151USP1), titled “LIGHTFIELDS,” by Domagoj Baričević, Vijay Kamarshi, Vijay Sundaram, filed on equalday, the contents of which are incorporated herein by reference in its entirety.

[0028] Each surface layer in the LSLF representation may be constructed or reorganized from camera-acquired information from one or more camera views. For example, some points on the surface layer may be visible in one or more specific camera views in the set of different camera views but not visible to one or more other camera views in the set of different camera views. Conversely, some points on the surface layer may be visible to the one or more other camera views but not the one or more specific camera views. These points visible from different camera views may be reorganized into the surface layer as the points are determined to be of implied connectivity on the same contiguous or continuous surface.

[0029] Each point in the collection of points constituting a surface layer as described herein carries a (e.g., precise, measured, determined, estimated, etc.) depth value, opacity (or transparency or occlusion) information such as an alpha channel or parameter (possibly view dependent; ranging from totally transparent to totally opaque or occluded), and color information such as (e.g., view dependent, etc.) component color values represented in an RGB or YUV space.

[0030] View dependent intensity (and / or color) mappings or values and occlusion information can be encoded by using spherical harmonics basis functions or corresponding coefficients. These spherical harmonics basis functions or corresponding coefficients can be used to describe view dependent variations at each point – with its occlusion information – which is incorporated in the multiple surface layers.

[0031] Occlusion information for a point represented in a surface layer of the multiple surface layers indicates whether the point is visible in one or more different views – which may be other (e.g., neighboring, etc.) camera views in the set of different camera views other than the target view – and whether the point is visible from the target view. For example, the occlusion information may indicate that the point is not visible from the target view, but the point is nevertheless visible in at least one other view for which intensity and / or color information is available. Hence, the intensity and / or color information and depth (or spatial positional) information of the point visible from the other view(s) may be transformed into the target view with the occlusion information indicating that the point is occluded or not visible from the target view. The intensity and / or color information of the point may not be used to render an image of the target view but may be used to synthesize or render in an image of a (e.g., novel, etc.) view – in which there is no occluding spatial layer in front of the point – different or reoriented from the target view. As illustrated in FIG.1B, this additional information, such as the intensity, color, depth and color information of the point, a translatedand reoriented virtual camera view in which the point is visible among other visible points, can be synthesized.

[0032] Image related quality of synthesized views may sometimes depend on the total number of (e.g., reference, non-reference, target, non-target, etc.) views used to construct or generate points constituting the multiple surface layers of the target view, as well as the total number of layers included or available in the LSLF set. The greater these total numbers are, the higher quality the synthesized views.

[0033] Also, larger field(s) of view covered by LSLF set(s) allow synthesis of larger translation extent by a recipient device of the LSLF set(s). However, improving quality and translation extent by sending a higher total number of target views and / or multiple surface layers tends to increase transmission bandwidth. This may be mitigated by creating multiple LSLF sets at different spatial locations or points in a 3D space including the scene and selecting and transmitting specific ones – or specific subsets of the multiple LSLF sets – closest to desired virtual (e.g., to be synthesized, novel, etc.) view as target views for which LSLF sets are transmitted. 3. DEPTH, TRANSPARENCY AND TEXTURE INFORMATION

[0034] A (spatial) point represented in an LSLF set as described herein may have some or all information or data values as follows: scene depth, radiance information, transparency (e.g., occlusion, opaqueness, etc.) information, etc.

[0035] The scene depth of the point may carry depth or corresponding parallax (or disparity) information. The farther the point (which may be a part of a visual object) is (or the larger the scene depth is), the less the parallax is (e.g., with the zero-parallax plane at infinity, etc.). In some operational scenarios, the scene depth or its corresponding parallax information can be represented or captured, for example, as a 16-bit depth value in the form of half float or as a fixed-point value depending on the depth range to be supported in these operational scenarios.

[0036] The radiance (or light ray) information of the point may represent or capture “view dependent” intensity and / or color information. The radiance information or its corresponding intensity and / or color information may be represented, captured or decomposed into spherical harmonics coefficients using spherical harmonics (basis) functions in each color component or channel of a color space such as a (R, G, B) or (Y, U, V) color space. In some operational scenarios, for each color component or channel, three orders of spherical harmonics (basis) functions with respective spherical harmonics coefficients may be used to represent the intensity or color information in that color component or channel,amounting to nine (9) spherical harmonics coefficients such as 12 bits per coefficient. Hence, 2712-bit numbers or coefficients – which may be subsampled and / or quantized and / or compressed and / or coded like 2D pixels or pixel values. One of the benefits from using spherical harmonics (basis) functions and coefficients for radiance information representation is to relatively accurately capture or represent intensity and color – e.g., specular light as opposed to diffusive light, etc. – that may visually or perceptually vary or change depending on specific view directions.

[0037] The transparency information of the point may be used to represent or capture occlusion, opaqueness, transparency, or translucency of the point or of the object to which the point belong in the scene. In some operational scenarios, the transparency information can be represented as a 12-bit alpha channel or parameter value.

[0038] In some operational scenarios, a three-dimensional or 3D (e.g., Cartesian, a non- Cartesian, etc.) coordinate system may be used to represent 3D locations – e.g., points represented in an LSLF set as described herein, etc. – in a scene or a physical volume space including the scene. By way of example but not limitation, a Z axis or dimension of the coordinate system may represent a target view or depth direction, whereas X and Y axes or dimensions of the coordinate system may represent a plane or a spherical section or a surface, for example corresponding to a field of view along the target view.

[0039] Zero, one or more points (or 3D locations) represented in the LSLF set may share the same 2D location or the same {X, Y} coordinate values. In some operational scenarios, multiple different points represented in the LSLF set may share or correspond to the same 2D location or {X, Y} coordinate values. Each of these different points may belong to a respective surface layer in multiple surface layers represented in the LSLF set. For example, a first point of the different points in a first layer of the multiple surface layers may carry foreground information (e.g., of an image, etc.) from the target view, whereas a second different point of the different point may carry occluded, translucent or background information (e.g., of the image, etc.) from the target view in a second different layer of the multiple surface layers. 4. BASIS FUNCTIONS AND COEFFICIENTS

[0040] Spherical harmonics (basis) function, or in short spherical harmonics (SH), are special functions defined with support on the surface of a sphere. In computer graphics or image processing operations as described herein, SH (functions) can be used as a tool to decompose, compose, encode or decode directional (e.g., intensity, color, color component orchannel, etc.) information denoted as ^(^, ^), where ^ represents elevation angle, ^represents azimuth angle, in a spherical representation as illustrated in FIG.1C.

[0041] The SH (basis) functions may be generally denoted as ^^^ (^, ^), where ^represents an order of a SH basis function, m represents an index of the SH basis function among all SH basis functions at the ^-th order. As SH basis functions define or form a harmonic series, there can be an infinite number of possible orders of the SH basis function.

[0042] For a given order ^, 2^ + 1 SH basis functions or their corresponding coefficientscan be selected or identified by the superscript or index ^ ∈ [−^, +^]. Therefore, up to thegiven order N, the total number of SH basis functions or their corresponding coefficientsfrom order 0 to order ^ is (^ + 1)^.

[0043] The more coefficients (or SH basis functions) are used, the more "directionalities" directional information coded with the coefficients (or SH basis functions) can be covered. A coefficient for order 0 – along with 0-th order SH basis function which may be a constant or fixed value such as one (1) – is a scalar (e.g., constant, etc.) and represents the "ambient term" or an average (or 0-th order) response of a (e.g., light emission, light reflection, etc.) signal with no specific direction or directionality. Coefficients for order 1 – along with the first order SH basis functions – represent the (first order) responses to the signal aligned with or in the directions of the X, Y and Z axes, respectively. Coefficients for order 2 – along with the second order SH basis functions – represent the (second order) responses to the signal with diagonal (or tensor) directions capturing or representing information or responses with relatively high spatial frequencies or relatively high directional details.

[0044] In some operational scenarios, coefficients and SH basis functions up to order 2 (9 coefficients) or order 3 (16 coefficients) per each color component or channel of a color space may be used to represent intensity and / or color information of a point represented in an LSLF set as described herein. For example, in operational scenarios in which the color space is RGB, coefficients for each unique combination of index and order among the SH basis functions may form a RGB float triplet. In some operational scenarios, in LSLF related image processing application, coefficients for order 0 may – or may be constrained to - be non- negative, while coefficients for higher order may be negative, zero (0) or positive. Additionally, optionally or alternatively, in some operational scenarios, artificial neural networks such as a LSLFNet (also referred to as SuRFNet interchangeably) may be designed or trained to predict SH coefficients and / or corresponding alpha (channel / parameter values) across different layers of an LSLF set. It should be noted that in various operational scenarios, specific limits on the total number of orders of SH basis functions may or may not be used. In some operational scenarios, higher orders as compared with the second or thirdorder SH basis functions may also be used to relatively accurately represent view dependent color or light information. In some operational scenarios, different or variable total numbers of orders of SH basis functions may be used for some or all layers in the LSLF set.

[0045] In some operational scenarios, each surface layer of multiple surface layers of a given LSLF set corresponding to – or temporally indexed with – a given time instance (or time step / point) incudes or contains the following data: a depth (e.g., exr, etc.) file carrying or encoded with depth information for some or all points of the surface layer; an alpha map (e.g., exr, etc.) file carrying or encoded with alpha values of the points of the surface layer; one or more SH coefficients images or (e.g., exr, etc.) files each of which may be an image or (e.g., exr, etc.) file carrying or encoded with SH coefficients for a particular combination of order or index among a set of SH basis functions used to represent directional or non- directional intensity and / or color (or reflectance in the case of light reflection point clouds) information of the points of the surface layer; etc.

[0046] For example, SH basis functions including and up to the second order may be used to carry intensity and / or color (or reflectance) information of points of the LSLF set. Hence, the one or more SH coefficients images or (e.g., exr, etc.) files of the surface layer of the LSLF set may include an SH0 (e.g., RGB, YUV, etc.) image or (e.g., exr, etc.) file carrying or encoded with the 0th order SH coefficients for an applicable color space, an SH1 (e.g., RGB, YUV, etc.) image or (e.g., exr, etc.) file carrying or encoded with the first index of the first order SH coefficients, … an SH8 (e.g., RGB, YUV, etc.) image or (e.g., exr, etc.) file carrying or encoded with the last index of the second order SH coefficients.

[0047] In some operational scenarios, all data (values) are stored as half floats in an image or (e.g., exr, etc.) file as described herein of the LSLF set. FIG.1D illustrates an example rendering of an LSLF set titled GuitarMan. The LSLF set includes or contains three surface layers as illustrated in FIG.1E through FIG.1G. As illustrated in FIG.1E, each surface layer of these surface layers may include or contain a depth image or (e.g., exr, etc.) file carrying or encoded with depth information of points of the surface layer. As illustrated in FIG.1F, each surface layer may also include or contain an alpha map image or (e.g., exr, etc.) file carrying or encoded with alpha (parameter / channel) values of the points of the surface layer. As illustrated in FIG.1G, each surface layer may also include or contain one or more images or (e.g., exr, etc.) files carrying or encoded with intensity and / or color information of the points of the surface layer.

[0048] In some operational scenarios, SH coefficients for SH basis functions including and up to a certain order such as two (2) may be carried or encoded in the image filescarrying or encoded with intensity and / or color information of the points of the surface layer. As illustrated in FIG.1H, one of these surface layers such as the middle surface layer of the FIG.1E through 1G may include nine (9) (e.g., RGB, YUV, etc.) images or (e.g., exr, etc.) files each of which include or contain SH coefficients of a specific combination of order and index for the SH basis functions in an applicable color space. 5. LSLF COMPRESSION OPERATIONS

[0049] FIG.2A illustrates an end-to-end LSLF compression workflow or process flow. Different surface lightfield layers in an LSLF set of a scene may have little correlation between or among one another in terms of their respective depth information, alpha information and / or intensity / color information. Without loss of generality, for simplicity, some or all LSLF processing and / or compression operations as illustrated in FIG.2A may be focused on one (e.g., single, etc.) layer use cases / scenarios in which the LSLF related operations can be performed with respect to a single surface layer. The same or similar LSLF related operations as illustrated in FIG.2A may be simply extended to or repeated for each of any other surface layer in the same LSLF set.

[0050] Compression schemes or related operations for LSLF may be implemented to operate with (or reuse) some or all available hardware 2D image codecs such as those planned to be or already deployed in the field or end user devices. All LSLF data – e.g., depth (of FIG.2A), alpha map (of FIG.2A), SH coefficients, etc. – may be treated as 2D image data and hence may be processed with these available codecs.

[0051] Block 204 of FIG.2A comprises receiving input or source LSLF data 202 for a surface layer in multiple layers of an LSLF set. The input or source LSLF data 202 for the surface layer includes a depth dataset, an alpha map dataset and a set of SH texture images.

[0052] Each of the depth dataset and alpha map dataset in the LSLF data 202 of FIG.2A can be treated as monochrome data or spatially varying intensity or luminance data represented by spatially varying Y values but with spatially constant chrominance Cb / Cr value(s) or number(s), thereby generating or constructing a corresponding monochrome image having three color components or channels of a color space.

[0053] SH coefficients – of an individual specific combination of order and index up to a specific order such as order 2 – for all color (e.g., RGB, YUV, etc.) channels or components of a selected or applicable color space may be collected or represented as an RGB image, YUV444 image, YUV420 image, or the like.

[0054] For example, a pre-processing tool (e.g., LSLFNet or SuRFNet, etc.) may be used to generate or pack the SH coefficients as texture information such as intensity and / or colorinformation of points of the surface layer into nine (9) SH texture (e.g., RGB, etc.) images (denoted as “SH0 texture (RGB)”, “SH1 texture (RGB)”, … “SH8 texture (RGB)” in FIG. 2A). Each of these SH texture images includes all SH coefficients of the points of the surface layer for all color components or channels (e.g., R / G / B, etc.) for a specific combination of order and index of the SH basis functions. These nine SH texture images may be alternatively referred to as SH0, SH1, …SH8. The SH coefficients packed in these texture images may be treated as image data such as RGB data, YUV data, etc., for the purpose of maximizing or enabling reuse of available 2D image codecs such as those deployed in the field or in end user devices.

[0055] As illustrated in FIG.2A, block 204 of FIG.2A further comprises performing preprocessing operation on the received LSLF data 202 or the depth, alpha map, SH texture images generated therefrom. These preprocessing operations may include, but are not necessarily limited to only, any, some or all of: data cleanup, converting data from float or half-float to integer, etc.

[0056] In some operational scenarios, the preprocessing tool such as SuRFNet may generate some data that may go beyond a specific (e.g., legal, allowed, valid, etc.) value range, especially for SH coefficients. For example, the SH0 texture image generated from the preprocessing tool may have value less than 0. Hence, in some operational scenarios, the data cleanup (or legalization) operations to be performed in block 204 may include two parts. First, for the SH0 coefficients or the SH0 texture image, all negative values generated by the preprocessing tool may be set to zero (0). Second, for SH1, SH2, … SH8, values out of valid (or legal) value ranges may be identified based on applicable (e.g., maximum, etc.) SH value thresholds. Any SH coefficients found to be outside the thresholds may be removed from, or set to zero (0) in, the SH coefficients or texture images. In some operational scenarios, in response to determining that a SH coefficient of a color component or channel at a spatial position is to be removed or set to zero (0), SH coefficients for all color components or channels at the same spatial position for all SH1 to SH8 coefficients and texture images are removed or set to zero (0). The thresholds or thresholding values may be determined or set by identifying outliers of SH1 to SH8 coefficients in a value or statistical distribution of these coefficients. For example, the thresholds or thresholding values may be determined or set that correspond to a valid or legal value range for SH coefficients in the form of a valid or legaldata percentile range [99.99, 0.01] for these SH coefficients. Outliers ^ may be found withthreshold operations (e.g., ^ℎ^ = ^^^(^ !(^)), etc.) using the valid or legal percentilerange. In various operational scenarios, the same or different thresholding values may be setor determined for SH1 to SH8 coefficients. Additionally, optionally or alternatively, transparency characteristics may be used to determine whether SH coefficients can be further “masked” (e.g., removed, etc.) based at least in part on the corresponding alpha map image with or without dilation. Example masking or dilation operations with respect to SH coefficients are described in U.S. Provisional Patent Application No.63 / 613,374, titled “TRANSMISSION OF VOLUMETRIC IMAGES IN MULTIPLANE IMAGING FORMAT,” by T. Lu, et al., filed 21 December 2023, the contents of which are incorporated by reference herein.

[0057] Image data (e.g., RGB or YUV data, etc.) in each of these texture images carrying corresponding SH coefficients may be converted from non-integer such as half float to integer. convert data from half float to integer. The total number of bits (or bit depth) per integer such as 10b or 12b may be specified or used in accordance with coding profiles supported by the image codecs.

[0058] In some operational scenarios, a relatively simple linear conversion is applied as follows. Minimum and maximum float values denoted as fMin and fMax may be first determined for data values represented in an image or (e.g., exr, etc.) file; 2). These minimum and maximum float values fMin and fMax map (fMin, fMax) may be mapped to corresponding minimum and maximum integer value denoted as (iMin, iMax). For example, if a 12b full value range is to be used by integer, then the minimum and maximum integer value iMin = 0, iMax=4095. For any float (input) value fIn, the output integer value denoted as iOut may be determined as follows: iOut = iMin + (fIn-fMin) / (fMax-fMin)*(iMax-iMin) (1)

[0059] The minimum and maximum float values (fMin, fMax) can be pre-fixed or can be signaled, for example as image or LSLF metadata, from an upstream device such as an encoder implementing blocks 204 through 208 to a downstream device such as a recipient decoder implementing blocks 208 through 212.

[0060] Additionally, optionally or alternatively, the float-to-integer conversion as described herein can also employs a non-linear mapping function, for example, gamma correction with the same or different gamma correction factors set or configured for different orders of SH coefficients. The non-linear mapping function may be implemented with other methods and / or models including but not limited to LUT, piece-wise polynomial, spline etc.

[0061] In some operational scenarios, color space conversion or transformation such as from RGB to YUV as well as subsampling operations may be performed or applied to generate or fit image data in a specific color space supported by the image codecs.Additionally, optionally or alternatively, in some operational scenarios, the preprocessing tool used to generate SH texture images may already generate or fit the image data in the specific color space and / or with the specific subsampling format without any further color space conversion or transformation and / or subsampling to allow the image codecs to operate in the specific color space and / or the specific subsampling format.

[0062] As illustrated in FIG.2A, block 206 may comprise converting the SH0 texture image represented in an input or intermediate color space such as an RGB color space into an intermediate SH0 image represented in a selected color space such as an YUV color space.

[0063] Additionally, optionally or alternatively, block 206 of FIG.2A may also comprise converting and / or packing the SH1-SH8 texture images into an SH texture image – in a selected color space, which may be the same as the color space in which the converted SH0 texture image is represented – with a subsampling format (e.g., denoted as “4x2” such as “442”, “422”, etc.).

[0064] The depth, alpha (map), SH0 texture images may all be relatively efficiently processed by the relatively widely available 2D image codecs deployed in the field or end user devices. As noted, for SH0 coefficients, if the original subsampling format or color space is those associated with the RGB color space, the SH0 coefficient data in these original subsampling format or color space can be converted to a different (or selected) color space or a different (or selected) subsampling format such as YUV 420 or 444 for better or improved coding efficiency by these image codecs. Additionally, optionally or alternatively, in some operational scenarios, to reduce the total number of the decoder (resource) instantiations, SH1 to SH8 coefficients may be packed into one overall SH texture image.

[0065] The SH1 to SH8 coefficients may only exist when there is non-Lambertian effect. Hence, these SH coefficients or data may be sparse, especially when compared with the SH0 coefficients. These SH coefficients or data may also have relatively small values as compared with the SH0 coefficients. For example, in the previous GuitarMan example as illustrated on the left in FIG.1G, SH coefficients in layer 0 contain mainly data in background. The percentage of zero value data in the SH0 coefficients in layer 0 is relatively high. However, in comparison, for SH1 coefficients in layer 0, only 6.7% coefficient are zero (0). To fix or ameliorate the non-SH0 data issue, relatively small coefficients for SH1 to SH8 may be set or reset to zero (0). For example, in response to determining that any coefficient is less that a minimum coefficient value threshold (e.g., 1e-3, or 10 to the power of -3, etc.), that coefficient may be set or reset to zero (0). As a result, the percentage of zero coefficients for SH1 in layer 0 can be increased from 6.7% to 60%.

[0066] To further reduce higher (or non-0th) order SH coefficients, additional data pruning method(s) can be applied. In one example, spatially isolated coefficients may be identified or located and set or reset to zero (0) at these spatial positions. Image quality or loss function value changes may be computed or determined or measured to determine whether these data pruning methods introduce relatively large errors or losses or image quality deteriorations. If that is the case, the data pruning or zero-out operations may be canceled or reversed.

[0067] Additionally, optionally or alternatively, in some operational scenarios, block 206 of FIG.2A may further comprise converting the monochrome depth and alpha (map) images represented in an input or intermediate color space into corresponding images represented in a selected color space such as the YUV color space supported by the image codecs.

[0068] In some operational scenarios, for multiple surface layers in an LSLF set as described herein, some or all images of the multiple surface layers of the same type – e.g., one of depth, alpha map, or a specific combination of order and index for SH coefficients, etc. – may be packed into one overall image, picture or atlas (e.g., with identifying information for distinguishing data of different surface layers, etc.).

[0069] In some operational scenarios, image codecs as described herein may support different coding layers such as base layer and enhancement layer(s). For example, the SH0 texture image can be carried or coded with the base layer or as a primary picture or image, whereas depth and alpha map images can be carried or coded with an enhancement layer or as an auxiliary picture or image.

[0070] Block 208 of FIG.2A comprises encoding LSLF data such as the depth, alpha map and SH coefficient or texture images into a coded bitstream to be delivered or transmitted directly or indirectly to downstream recipient / decoding device(s). The coded bitstream may also be encoded or used to include, carry or signal LSLF metadata specifying operational parameters of image processing operations including but not limited to LSLF reconstruction operations to be performed by the downstream recipient / decoding device(s). The LSLF data and metadata may be encoded or carried as (coding) syntax elements in the coded bitstream in accordance with one or more coding specifications or (coding) syntaxes defined or specified therein.

[0071] On the decoder side, as illustrated in FIG.2A, block 210 comprises decoding, by the downstream recipient device from the received bitstream, the depth, alpha map and SH coefficient or texture images.

[0072] Block 212 of FIG.2A comprises unpacking, by the downstream device, SHcoefficients represented in the decoded SH coefficient images. In some operational scenarios, block 212 further comprises performing color space conversion or transformation operations from a selected color space (e.g., YUV, etc.) in which the decoded (e.g., SH0, etc.) image data is represented to a target color space (e.g., RGB, etc.).

[0073] Block 214 of FIG.2A comprises performing, by the downstream device, data conversion operations including but not necessarily limited to only float to integer conversions on the decoded image data received with the coded bitstream to generate reconstructed depth data, reconstructed alpha map data, and reconstructed (or rec) SH coefficient data representing texture information such as direction-dependent (e.g., specular reflection, etc.) or non-direction-dependent (e.g., diffusive reflection, etc.) intensity and / color data associated with the points of the multiple surface layers of the input or original LSLF set used to generate the coded bitstream.

[0074] In some operational scenarios, the upstream device may perform an optimization step in block 214 as a part of generating to-be-encoded data for the coded bitstream. The upstream device may perform some or all of the decoding operations such as blocks 212 and 214 of FIG.2A to generate the reconstructed LSLF data set 216 on the decoder side. The upstream device can compare image or LSLF qualities between the input or original LSLF set 202 and the reconstructed LSLF based on one or more errors – used to measure image or LSLF quality – computed with one or more selected error / loss functions. The to-be-encoded data may be generated or optimized by minimizing the errors or quality differences between the input or original LSLF set 202 and the reconstructed LSLF 216. Hence, in these operational scenarios, what the downstream device receives by way of the coded bitstream represents an optimized version of the reconstructed LSLF set 216 relatively closely approximating the image or LSLF quality of the input or original LSLF set 202. 6. HIGHER ORDER COEFFICIENT PACKING

[0075] In some operational scenarios, higher order SH coefficients – e.g., other than the 0th order SH coefficients – may be packed together in one image, frame or atlas using one or more syntaxes and / or syntax elements in accordance with the applicable coding specifications.

[0076] Example syntaxes or syntax elements relating to coding or packing these relatively high order SH coefficients are illustrated in TABLE 1 below. TABLE 1SH_pack_information( ) { Descripto r, _ _ _ _ _ be used to specify the width, in units of luma samples, of a packed (e.g., decoded, pre-encoded, etc.) picture / image used to contain or include the SH coefficients, may not be equal to 0, and may be an integer multiple of Max(8, MinCbSizeY), where Max() is a function that selects the larger of the two arguments as output.

[0078] The syntax element “packed_pic_height_in_luma_samples” may be used to specify the height, in units of luma samples, of the packed decoded picture / image, may not be equal to 0, and may be an integer multiple of Max(8, MinCbSizeY).

[0079] The syntax element “org_pic_width_in_luma_samples” may be used to specify the width, in units of luma samples, of an original high order SH coefficient image / picture (e.g., one of SH1, … SH8 in 202 of FIG.2A, etc.) to be packed into the packed decoded picture / image, and may not be equal to 0.

[0080] The syntax element “org_pic_height_in_luma_samples” may be used to specify the height, in units of luma samples, of the original high order SH coefficient image / picture, and may not be equal to 0.

[0081] The syntax elements “fMin[ j ]” and “fMax[ j ]” may be used to specify the minimum and maximum values of the j-th order SH coefficients in the unit of 0.00001 in the original high order SH coefficient image / picture, wherein j represents an integer no less than one (1).

[0082] FIG.1I illustrates an example packed decoded picture / image – which may also be referred to as (packed) atlas – into which relatively high order SH coefficient images / pictures are packed. The example atlas includes or contains three (3) surface layer SH1-SH8coefficient pictures / images (totaling 24 pictures / images) packed into the atlas in a 6x4 spatial pattern or array. Gray color in the atlas indicates coefficients with zero (0) values. As illustrated, the SH1-SH8 data or coefficients are relatively sparse with most picture / image / atlas portions containing zero value SH coefficients or data.

[0083] To use packing more efficiently, therefore making techniques as described herein more easily implemented by a wide variety of available codecs or decoders, some or all example V3C (MPEG Visual Volumetric Video-based Coding) and MIV (MPEG Immersive video) atlas packing operations or components – such as described in U.S. Provisional Patent Application No.63 / 191,480, titled “PATCH-BASED RESHAPING AND METADATA FOR VOLUMETRIC VIDEO,” by Guan-Ming Su and Peng Yin, filed on 21 May 2021, the entire contents of which are incorporated by reference herein – may be used or implemented.

[0084] The atlas or image / picture may be partitioned into non-overlapping patches. Patches with nonzero coefficients or data may be referred to as significant patches. A new atlas or image / picture – to be encoded into the coded bitstream – may be formed with all significant patches (e.g., only, without any other patches that are not significant, etc.) in the (existing or input) atlas into which the relatively high order SH coefficient images / pictures are packed.

[0085] Example syntaxes or syntax elements relating to coding the significant patches identified or determined from the (input) atlas are illustrated in TABLE 2 below. A syntax table including these syntaxes or syntax elements may be inserted in a relatively high level syntax carried by the coded bitstream such as video parameter set (VPS), sequence parameter set (SPS), picture parameter set (PPS), picture header (PH), slice header (SH), and / or supplemental enhancement information (SEI) messages. TABLE 2 SH_atlas_pack_information( ) { Descriptofor( i = 0; i < = num_layers_minus1; i++ ) for( j = 0; j < numHighOrderSH; j++ ) / *code high order coeffs* / {

[0086] n , e syn ax e emen pac ed_p c_w d _ n_ uma_samp es may be used to specify the width, in units of luma samples, of the packed decoded picture (or input atlas), may not be equal to 0, and may be an integer multiple of Max(8, MinCbSizeY).

[0087] The syntax element “packed_pic_height_in_luma_samples” may be used to specify the height, in units of luma samples, of the packed decoded picture (or input atlas), may not be equal to 0, and may be an integer multiple of Max(8, MinCbSizeY).

[0088] The syntax element “org_pic_width_in_luma_samples” may be used to specify the width, in units of luma samples, of an original high order SH coefficient image / picture (e.g., one of SH1, … SH8 in 202 of FIG.2A, etc.) to be packed into the packed decoded picture / image (or the input atlas), and may not be equal to 0.

[0089] The syntax element “org_pic_height_in_luma_samples” may be used to specify the height, in units of luma samples, of the original high order SH coefficient image / picture, and may not be equal to 0.

[0090] The syntax element “min_patch_size_minus1” plus 1 may be used to specify the minimum patch size “minPatchSize” (or a basic unit of patch size). A patch size as described herein may be multiple (including one (1)) of the minimum patch size.

[0091] The syntax element “num_layers_minus1” plus 1 may be used to specify thenumber of surface layers for SH coefficients.

[0092] The syntax element “num_SH_order_minus1” plus 1 may be used to specify the number of orders for SH coefficients.

[0093] The syntax element “num_patches_minus1[ i ][ j ]” plus 1 may be used to specify the number of patches in the packed picture or input atlas for the i-th layer, (j+1)-th SH (higher than the 0th) order coefficients, where each of i and j represents a non-negative integer.

[0094] The syntax element “patch_x_coor[ i ][ j ][ k ]” may be used to specify the x coordinate in the packed picture or atlas for the i-th layer, (j+1)-th SH (higher than the 0th) order coefficients and k-th patch, where k represents a non-negative integer.

[0095] The syntax element “patch_y_coor[ i ][ j ][ k ]” may be used to specify the y coordinate in the packed picture or input atlas for the i-th layer, (j+1)-th SH (higher than the 0th) order coefficients and k-th patch.

[0096] The syntax element “org_patch_x_coor[ i ][ j ][ k ]” may be used to specify the x coordinate in an original i-th layer, (j+1)-th SH (higher than the 0th) order coefficient picture and k-th patch.

[0097] The syntax element “org_patch_y_coor[ i ][ j ][ k ]” may be used to specify the y coordinate in the original i-th layer, (j+1)-th SH (higher than the 0th) order coefficient picture and k-th patch.

[0098] The syntax element “patch_width_mul_minus1[ i ][ j ]” plus 1 then multiplied with “minPatchSize” may be used to specify the width of the patch for the i-th layer, (j+1)-th SH (higher than the 0th) order coefficient picture and k-th patch.

[0099] The syntax element “patch_height_mul_minus1[ i ][ j ]” plus 1 then multiplied with “minPatchSize” may be used to specify the height of the patch for the i-th layer, (j+1)-th SH (higher than the 0th) order coefficient picture and k-th patch.

[0100] At the decoder side, a downstream recipient device of the coded bitstream may decode the significant patches from the coded bitstream and arrange the significant patches as well as non-significant patches not explicitly encoded in the coded bitstream back to their respective original positions in the input atlas into which the relatively high order SH coefficient images / pictures are packed.

[0101] On the decoder side, by way of the data structures as illustrated in TABLE 2 coded in the coded bitstream, a downstream recipient device can determine the total number of the patches with the explicitly signaled “num_patches_minus1” or implicitly count the total number of received / decoded patches.

[0102] The downstream device can also determine the location or position (e.g., within the original or input atlas used to pack both significant patches and non-significant patches, etc.) of each patch with the (x, y) coordinates explicitly signaled by the syntax elements (patch_x_coor, patch_y_coor).

[0103] The downstream device can further determine the width and height of the patch with the signaled size (patch_width, patch_height), the original location or position (e.g., within a field of view of the target view represented by the LSLF set, in a Cartesian coordinate system of the scene, etc.) of the patch with the (x, y) coordinates explicitly signaled by the syntax elements (orig_patch_x_coor, orig_patch_y_coor), the total number of surface layers signaled by the syntax element (num_layers_minus1), and the total number of SH orders signaled by the syntax element (num_SH_orders_minus1).

[0104] The patch size and / or coordinates values can be actual or absolute values or a relative or re-calculated value such as by dividing with a common ratio or division factor, in the latter of which a syntax element or value (min_patch_size) may be used to signal – by the upstream encoding device to the downstream decoding device – the ratio or division factor. The downstream device can recover or restore the original actual or absolute values of the patch coordinates and sizes by way of multiplying with the common ratio or division factor.

[0105] All those syntaxes or syntax elements can be coded in the bitstream, for example, in the relatively high level syntax (HLS), which may be one or more of: VPS, SPS, PPS, PH, SH, SEI, etc. In some operational scenarios, for temporal consistency, the packing information as signaled with the syntaxes or syntax elements illustrated in one or both of TABLES 1 and 2 remain fixed or consistent within a coded video sequence in one or more coded video sequences of the coded bitstream.

[0106] An example procedure or process flow for forming a new atlas or picture including (e.g., all, only, etc.) significant patches for multiple surface layers for all SH1 to SH8 coefficients in the LSLF set is illustrated in TABLE 3 below. TABLE 3 code num_layers_minus1 code num SH orders minus1 (note: #of coefficients is n^2) ht,

[0107] In various operational scenarios, (e.g., significant, etc.) patches signaled with the atlas (e.g., comprising significant patches, etc.) in the coded bitstream may be collectively or individually scaled. Additionally, optionally or alternatively, different (e.g., significant, etc.) patches signaled with the atlas (e.g., comprising significant patches, etc.) in the coded bitstream may have, or may be assigned to, different spatial orientations and / or different spatial locations / positions. These patches may be spatially transformed (e.g., rotation, translation, etc.) to different spatial orientations and / or different spatial locations / positions, for example, to improve packing efficiency or reduce bitrate usage. 7. CODING BLOCK BASED COMPRESSION

[0108] In some operational scenarios, images / atlases comprising depth information / data, alpha map information / data and SH0 coefficients can be encoder-side processed, encoded, transmitted, decoded, decoder-side processed or rendered relatively efficiently with a wide variety of available image codecs deployed in the field or in end user devices. In comparison, images / atlases comprising (e.g., packed, etc.) SH1 to SH8 coefficients – even though sparse – takes or consumes a relatively high bitrate in the coded bitstream. In some operational scenarios, one or more methods as described herein may be specifically used to compress these higher order SH coefficients.

[0109] A first (higher order than the 0th order; example) SH coefficient compression method may be implemented or performed – by one or more image codecs – to directly and independently code non-zero order SH coefficients (e.g., SH1 - SH8, etc.). As non-zero order SH coefficients are sparse, different spatial positions and / or different levels (or level values) of non-zero SH coefficients may be coded separately. By way of illustration but not limitation, SH1 coefficients may be compressed with the first compression method as an example. The same compression methods performed for the first higher order SH1 may be (e.g., separately, independently, etc.) performed or repeated for other higher orders such as SH2-SH8.

[0110] Non-zero SH coefficients may be referred to as significant coefficient. Respective positions of significant coefficient may be specified in or as a significance map. To reduce bitrate usage for coding non-zero coefficient positions, an SH1 image – or a patch, a significant patch, etc. – comprising the significant coefficients at their respective positions may be spatially partitioned or divided into (e.g., mutually exclusive, etc.) CTBs (coding tree blocks).

[0111] Each CTB – which may be a square as illustrated in FIG.3A (a) – may be further partitioned (e.g., as specified in AVC, HEVC, VVC, etc.) into CBs (coding blocks) forming aquadtree structure as illustrated in FIG.3A (b) with the CTB as the root. An example CTB (e.g., squared, etc.) size may be up to 64x64 pixels by pixels (or positions by positions). In various operational scenarios, a CTB as described herein may be represented by a single CB or may be split into four partitions of half horizontal and half vertical size. Each of these partitions or subdivisions of the CTP may be a CB itself, which may or may not be further partitioned to half horizontal and half vertical sizes, and so forth. In some operational scenarios, the minimum coding block (e.g., squared, etc.) size is 8x8 pixels by pixels (or positions by positions).

[0112] For every partition or CB, (e.g., HEVC, etc.) syntax elements denoted as “cbf_luma, cbf_cb, cbf_cr” may be coded or signaled in the coded bitstream for corresponding color components or channels of the selected color space for a current coding block in which non-zero SH1 coefficients are present or included. Using luma as example, cbf_luma with a value of one (1) may be used to indicate or signal that the current coding block contains one or more SH1 coefficients not equal to 0 in the luma color channel or component. Similarly, cbf_cb with a value of one (1) may be used to indicate or signal that the current coding block contains one or more SH1 coefficients not equal to 0 in the Cb color channel or component; cbf_cr with a value of one (1) may be used to indicate or signal that the current coding block contains one or more SH1 coefficients not equal to 0 in the Cr color channel or component.

[0113] For any coding block with a cbf value not equal to 0, non-zero coefficient positions may be signaled by way of a (e.g., AVC, HEVC, VVC, etc.) significance map. In some operational scenarios, multi-level (e.g., HEVC, etc.) significance signaling may be employed or implemented under techniques as described herein. The current coding block such as a 16x16 CB may be split into 4x4 sub-blocks (or coefficient groups) as illustrated in FIG.3B. First, coordinates (horizontal and vertical) of the last 4x4 sub-block – in the current coding block as illustrated in FIG.3B – having at least one non-zero coefficient are coded or signaled in the bitstream. Then for each 4x4 sub-block preceding the last 4x4 sub-block, a coded sub-block flag or syntax element denoted as “coded_sub_block_flag” may be coded or used to signal whether the (current preceding) sub-block contains at least one nonzero coefficient. In the case in which the “coded_sub_block_flag” equals one (1), a significant flag or syntax element denoted as “sig_coeff_flag” is signaled or coded for each coefficient prior to the indicated position of the last significant coefficient.

[0114] Coefficient groups such as those in 4x4 sub-blocks of the coding block may be processed sequentially in the reverse order (from bottom-right to top-left) as illustrated inFIG.3C. Within one coefficient group, the scanning of coefficients can start with the last significant coefficient and proceeds to the DC (e.g., with the coordinates of 0x0 in the sub- block, etc.) coefficient in the reverse scanning order such as along a diagonal order, a horizontal order, a vertical order, etc., as illustrated in FIG.3C.

[0115] For each position / location (e.g., in a sub-block, etc.) at which the corresponding sig_coeff_flag is equal to one, the coefficient level may be coded in accordance with one or more applicable coding specifications such as those relating to AVC, HEVC, VVC, etc. For example, coefficient greater than one or two flags specifying whether the level value is greater than one or two may be first coded. The (e.g., remaining, etc.) level value may also be coded depending on the two specific values. The sign of the coefficient(s) may further be coded. Syntaxes and / or syntax elements as described herein may be coded using entropy coding, for example, CABAC (Context-based adaptive binary arithmetic coding). The context in the entropy coding can be dependent on (e.g., coefficient, etc.) position and / or its spatial neighbors.

[0116] Original or input SH1 coefficients may be float values, which may be converted to corresponding integer values by quantization. Both linear and non-linear quantization methods can be applied. Additionally, optionally or alternatively, relatively advanced reshaping methods may be used to map the SH1 coefficients into mapped values coded in the bitstream. The same or different quantization parameters may be specified collectively or individually for different partitions, blocks, or sub-blocks. The quantization parameters can be predicted using data or parameter values applied to spatial neighbors. At the decoder side, inverse quantization and other corresponding decoder-side operations can be applied.

[0117] A second (higher order than the 0th order; example) SH coefficient compression method may be implemented or performed – by one or more image codecs – to jointly code non-zero order SH coefficients (e.g., SH1 - SH8, etc.). For example, the first (higher order than the 0th order) SH coefficient compression method or approach can be extended to three- dimensional (3D) by concatenating SH1 to SH8 coefficients in or along a third dimension. The position coding therefore may be extended to one additional dimension to the two dimensions previously handled by the pre-extended first (higher order than the 0th order) SH coefficient compression method or approach. Quantization / dequantization operations in the pre-extended first method may also be extended from scalar to vector quantization / dequantization.

[0118] In some operational scenarios, some or all SH1-SH8 coefficients may be treated as neural networks (NN) coefficients. Neural Network Compression (NNC) operationscorresponding to a combination of quantization and deepCABAC operations may be directly applied by way of artificial neural networks (ANNs) in accordance with one or more applicable specifications or standards, an example of which may be “MPEG-7: Compressed representation of neural networks for multimedia content description and analysis: ISO / IEC 15938-17” incorporated by reference herein in its entirety.

[0119] In some operational scenarios, (intra and / or inter) prediction operations may be applied to remove redundancy of SH coefficients in temporal (and / or spatial) domain(s). Prediction operations for SH1 to SH8 coefficients may share, or may be performed in, the same mode (intra or inter) as SH0 coefficients in at least some of the prediction operations.

[0120] For example, in response to determining that SH0 coefficients is coded in an inter- picture prediction mode, SH1 to SH8 coefficients are coded in the same inter-picture prediction mode. Additionally, optionally or alternatively, SH1 to SH8 coefficients prediction operations can share the same partition and mode syntaxes or syntax element as those of SH0 coefficients. As a result, explicit signaling for these syntaxes or syntax elements may be avoided or reduced for SH1 to SH8 coefficients.

[0121] On the other hand, in response to determining that SH0 coefficients is coded in an intra prediction mode, the first higher order coefficient compression method (e.g., directly coding non-zero order SH coefficients without partition signaling, etc.) may be used.

[0122] If the inter-prediction mode is used, the same motion vector (MV) information or data as that of SH0 may be used for the inter-prediction operations relating to the SH1 to SH8 coefficients. A residue signal comprising residual SH1 to SH8 coefficients (or coefficient values) may be generated based at least in part on motion compensation. The residue signal may be coded with the first higher order coefficient compression method (e.g., directly coding non-zero order SH coefficients without partition signaling, etc.).

[0123] One of many benefits of using the coding methods as described herein is for relatively easy implementation by reusing or modifying coding components supported by a wide variety of image codecs deployed in the field or end user devices. 8. POINT CLOUD COMPRESSION

[0124] In some operational scenarios, a third (higher order than the 0th order; example) SH coefficient compression method may be implemented or performed by way of point cloud compression (PCC). For example, instead of coding or compressing SH1 to SH8 coefficients corresponding to significant (or non-zero) coefficient positions independently, these SH1 to SH8 coefficients may be packed or arranged or represented along a third dimension (e.g., representing values of 1 to 8 respectively used to label the SH1 to SH8 coefficients, etc.). Thesignificant (or non-zero) coefficient positions – corresponding to these (non-zero) SH1 to SH8 coefficients – in a 3D space constructed based in part on the third dimension denoted as Z (along with X, Y locations / positions of a field of view in reference to the target view) form or generate a specific geometry for a point cloud in the 3D space. Geometry-based coding operations may be adapted, implemented or used to code the SH1 to SH8 coefficients, jointly, to which the significant points of the point cloud correspond. Example geometry-based coding operations are described in M. Quach, J. Pang, D. Tian, G Valenzise, and F. Dufaux, “Survey on Deep Learning-Based Point Cloud Compression”, Frontiers in Signal Processing (February 23, 2022), the entire contents of which are incorporated by reference herein.

[0125] The point cloud in the Z dimension / axis (or domain) may be vectorized or valued from 0 to 7 for the SH1 to SH8 coefficients. Hence, Z equal to 0 corresponds a first X-Y plane of the 3D space in which the SH1 coefficients or their corresponding significant points reside; Z equal to 1 corresponds a second X-Y plane of the 3D space in which the SH2 coefficients or their corresponding significant points reside; …; Z equal to 7 corresponds an eighth X-Y plane of the 3D space in which the SH8 coefficients or their corresponding significant points reside.

[0126] Hence, for every significant (or non-zero) SH coefficient, there is a corresponding significant point with specific (x, y, z) coordinates among the significant points forming the point cloud. The problem of coding significant coefficient positions and their corresponding SH coefficients becomes that of geometry coding in G-PCC as illustrated in FIG.2B.

[0127] More specifically, the G-PCC or geometry-based coding operations of FIG.2B comprises receiving, by an upstream encoding device, the (input or original) geometry of the point cloud and their (e.g., 1-1 mapping, etc.) corresponding (input or original) SH coefficients (denoted as “Attributes” in FIG.2B). Geometry compression operations are performed by the upstream device on the (input or original) geometry to generate compressed geometry to be encoded in a coded bitstream from the upstream device and to be decoded by a downstream decoding device of the coded bitstream. Geometry decompression operations – which may be the same as those to be performed by the downstream device – may be performed by the upstream device on the compressed geometry to generate a decompressed geometry of the point cloud. The decompressed geometry – which may comprise a plurality of locations approximating the (input or original; pre-compressed) point cloud – may be the same as that to be generated by the downstream device, subject to any coding or data transmission errors. The (input or original) attributes or SH coefficients may be transferred or mapped (e.g., interpolation, extrapolation, linear or non-linear mapping, etc) to generate (e.g.,1:1 mapping) corresponding transferred attributes or transferred SH coefficients for the decompressed geometry or positions defined or specified therein. Attribute compression operations may be performed by the upstream device on the transferred attributes or transferred SH coefficients for the decompressed geometry to generate compressed attributes or compressed SH coefficients, which can be encoded in the coded bitstream along with the compressed geometry of the point cloud. On the decoder side, attribute decompression operations can be performed by the downstream device on the compressed attributes or compressed SH coefficients of the coded bitstream to generate decompressed attributes or decompressed SH coefficients (e.g., 1:1 mapping, etc.) corresponding to the decompressed significant locations forming the decompressed geometry of the point cloud.

[0128] In some operational scenarios, geometry point cloud compression (or G-PCC) coding techniques, as illustrated in FIG.2C may be applied to carry out the geometry coding or compression operations of a point cloud as described herein. Example G-PCC operations are described in ISO / IEC DIS 23090-9, “Information technology — Coded representation of immersive media: Part 9— Geometry-based point cloud compression,” which is incorporated by reference herein in its entirety.

[0129] Under a first example approach, an octree structure may be used to represent voxelized points (or voxels) generated from the point cloud. As used herein, voxelized points may refer to grid points of a (e.g., regular, uniformly sampled, etc.) 3D grid or spatial array distributed in the 3D space in which the point cloud resides. The significant locations / positions forming the point cloud in the 3D space may be used to generate the voxelized points or attributes or SH coefficients for these points, for example through interpolation or extrapolation operations. The (3D) octree structure – which may be analogous to (2D) quadtree – may be used to represent the voxelized points with corresponding attribute values generated from the significant locations or positions and corresponding SH coefficients of the point cloud. A wide variety of techniques may be used to perform relatively efficient G-PCC operations on the octree.

[0130] Under a second example approach, a point-by-point predictive geometry method. For example, a prediction model (e.g., artificial neural networks, transformer networks, multi-level perceptron or MLP networks, etc.) may be trained to operate with optimized or trained operational parameter values to generate a predicted geometry – approximating the input or original geometry – of the point cloud and / or attributes or SH coefficients for significant locations / positions in the predicted geometry.

[0131] Under a third example approach, a trisoup representation may be used represent the point cloud or the (input or original) geometry thereof. The trisoup representation represents an object surface, for example in the 3D space such as generated from the point cloud, as a series of triangle meshes. This approach tends to work well for a dense surface point cloud as compared with sparse surface point. On the encoder side, the upstream device can apply mesh coding operations to the trisoup representation or the triangle meshes therein to generate a compressed trisoup representation or data to be encoded in a coded bitstream. On the decoder side, a downstream recipient device of the coded bitstream can generate a reconstructed point cloud from the mesh surface or triangle meshes – e.g., in a specific voxel or grid granularity that ensures a target density of the reconstructed point cloud.

[0132] For these approaches, context based entropy coding operations may be applied in coding compressed data of the coded bitstream. Based on characteristics of SH1 to SH8 coefficients, one of these approaches / methods or the like may be selected to code the point cloud geometry information along with the corresponding attribute or SH coefficient information.

[0133] In addition to these approaches / methods relating to G-PCC, in some operational scenarios, graph transform may be applied for the purpose of carrying out geometry compression. Example graph transforms are described in the previously mentioned Quach et al., “Survey on Deep Learning-Based Point Cloud Compression”, Frontiers in Signal Processing (February 23, 2022). Additionally, optionally or alternatively, some or all of the geometry compression operations may be performed on geometry and / or attribute (SH coefficient) data in a spectral domain other than or in addition to a pixel domain.

[0134] FIG.2C illustrates example encoder-side G-PCC compression operations that include a combination of geometry coding and attribute (or SH coefficient) coding based at least in part on octree, trisoup (or surface meshes), etc. More specifically, the G-PCC of FIG. 2C comprises receiving, by an upstream encoding device, the input or original (e.g., source, pre-compressed, etc.) point cloud (denoted as “Point Cloud” in FIG.2C) including the input or original (e.g., source, pre-compressed, etc.) geometry (denoted as “Geometry” in FIG.2C) of the point cloud and their (e.g., 1-1 mapping, etc.) corresponding input or original or pre- compressed SH coefficients (denoted as “Attributes” in FIG.2C).

[0135] For geometry coding, the G-PCC of FIG.2C comprises performing, by the upstream device, coordinate transformation and voxelization operations on the input or original geometry of the point cloud. These coordinate transformation and voxelization operations transform, convert and / or voxelize (e.g., significant, etc.)points / locations / positions in the input or original geometry of the point cloud into voxels and an octree of 3D sub-blocks in a 3D grid or spatial array spanning a 3D space that contains the scene represented in the point cloud.

[0136] The G-PCC of FIG. 2C comprises performing, by the upstream device, geometry analysis operations on the voxels or octree using an octree and / or trisoup scheme. In some operational scenarios, a geometry of the point cloud after the coordinate transformation and voxelization may be a pruned octree (e.g., free of unoccupied sub-blocks, etc.) in which some or all leaves represent (e.g., significant point / location / position, etc.) occupied sub-blocks larger than a voxel. The geometry as represented by the pruned octree may be reduced or transformed into a surface model with one or more object surfaces approximated by a series of triangles or trisoup (“triangle soup”).

[0137] Arithmetic coding or coder may be applied or invoked by the upstream device to compress the trisoup or octree into compressed geometry encoded into a coded bitstream.

[0138] For attribute or SH coefficient coding, the G-PCC of FIG.2C comprises performing, by the upstream device, color space operations on the attributes or SH1-SH8 coefficients of the point cloud in operational scenarios in which a different color space is to be used to represent the attributes or SH coefficients.

[0139] The G-PCC of FIG.2C comprises performing a number of transform operations, which may include any, some or all of: a Region Adaptive Hierarchical Transform (RAHT), a Predicting Transform, a Lifting Transform, etc. RAHT may be implemented to use attribute values in a lower octree level as input to predict or generate attribute values in the next octree (e.g., higher, etc.) level as output. The Predicting Transform may be used to implement an interpolation-based hierarchical nearest-neighbor prediction scheme. The Lifting Transform may be built on top of the Predicting Transform with an extra update / lifting step. Example operations relating to RAHT, Predicting Transform or Lifting Transform are described in the previously mentioned ISO / IEC DIS 23090-9, “Information technology — Coded representation of immersive media: Part 9— Geometry-based point cloud compression.”

[0140] Quantization (or quantizer) and arithmetic coding (or coder) may be applied (or invoked) by the upstream device to compress transformed attributes or SH coefficient data generated from the transform operations into compressed attributes or SH coefficient data encoded into the coded bitstream with the compressed geometry of the point cloud. 9. NEURAL NETWORK BASED GEOMETRY CODING

[0141] AI based technologies such as neural network based geometry coding may be used to compress point cloud geometry or SH efficient data represented in a point cloud. ExampleAI-based point cloud compression (PCC) operations are described in the previously mentioned Quach et al., “Survey on Deep Learning-Based Point Cloud Compression”, Frontiers in Signal Processing (February 23, 2022). As high order (or other than the 0th order) SH coefficients tend to be sparse, point-based AI-based approaches may have better performance than those voxel grid based AI-based approaches. In some operational scenarios, convolutional neural network (CNN) based autoencoders may be used in point cloud geometry (and / or SH coefficient data) coding. CNN based methods using sparse convolution may be used to achieve better tradeoff on complexity versus compression performance than other AI-based or non-AI based approaches. 10. MESH CODING

[0142] Meshes can be constructed from point clouds, for example using Poisson surface reconstruction. Point clouds can be sampled from or represented with the meshes. In some operational scenarios, the percentage of non-zero SH high order coefficients are relatively high. In these operational scenarios, the meshes can be encoded directly. Example (e.g., MPEG, video based dynamic, etc.) mesh coding operations are described in “MPEG output document: WG07 N00680: Information technology — Coded representation of immersive media — Part 29: Video-based dynamic mesh coding (V-DMC): WD4.0,” which is incorporated by reference herein in its entirety.

[0143] In some operational scenarios, a coded bitstream may include mesh geometry bitstreams as (e.g., two, etc.) sub-bitstream. One of the mesh geometry bitstreams or sub- bitstreams is encoded with base mesh geometry, whereas another of the mesh geometry bitstreams or sub-bitstreams is encoded with displacement geometry refinement. The base mesh sub-bitstream may be coded or decoded using two components: a static mesh coder or decoder indicated by or implemented with a first V3C component, and / or a motion vector field coder or decoder indicated by or implemented with a second V3C component. The motion vector field may be (e.g., only, etc.) coded or decoded for registered meshes. The displacement component or geometry refinement may be coded or decoded using a codec indicated by or implemented with a third V3C component. Operational parameters such as coefficients of the displacement component or geometry refinement may be dequantized and converted to displacement values using an inverse wavelet transform. 11. FLOATING POINT PLANE BASED COMPRESSION

[0144] In some operational scenarios, prediction, quantization, entropy coding, etc. – such as those used in video coding – may be applied on implemented to code (e.g., higher order, non-0th order, etc.) SH coefficients. All these SH coefficients planes – each of whichrepresents a specific set among (e.g., eight different, etc.) sets of SH1-SH8 coefficients – into a 3D SH cube. A Lorenzo predictor (3D prediction) may be implemented to generate residual (SH coefficient data). The residual may be quantized from float point values to integer values. Arithmetic coding may be implemented or performed to code the quantized residual into a coded bitstream to be delivered from an upstream device to a downstream recipient device. Example Lorenzo predictor or 3D prediction operations are described in Peter Lindstrom and Martin Isenburg, “Fast and Efficient Compression of Floating-Point Data,” IEEE Transactions on Visualization and Computer Graphics (September 2006), which is incorporated by reference in its entirety.

[0145] Additionally, optionally or alternatively, in some operational scenarios, other or relatively sophisticated 3D prediction method(s) other than Lorenzo prediction can be built – with flexibility for or depending on – mode decisions and / or characteristics or distributions of SH coefficient data. Also, one or more different scanning orders for processing SH coefficients with respect to the 3D SH cube or planes therein can be designed or implemented. 12. TRANSFORM DOMAIN 3D SH CUBE

[0146] A 3D unitary block / cube (4x4x4) transform can be applied to convert SH coefficients (e.g., in a spatial domain in which the SH coefficients are distributed, etc.) to transformed SH coefficients in another transform such as a frequency domain. Quantization – such as described in Peter Lindstrom, "Fixed-Rate Compressed Floating-Point Arrays", IEEE Transactions on Visualization and Computer Graphics (August 2014), which is incorporated by reference herein in its entirety – may be performed on the transformed coefficients in the transform domain, followed by entropy coding into a coded bitstream.

[0147] An example separate 1D transform – which may be used to perform domain transformation operations – is given as follows: (2) where tbe performed with respect to this transform or other transforms for different specific values of t, for example, to determine which of the transforms is to be applied for the domain transformation operations.

[0148] Additionally, optionally or alternatively, in some operational scenarios, transform domain compression may be further improved by adding or performing 3D prediction in addition to or in combination with the domain transformation, quantization and / or arithmetic coding operations. 13. SPARSE MATRIX, HDF5 AND SPATIAL FREQUENCY

[0149] As quantized SH planes can be sparse with non-zero SH coefficients. A sparse matrix compression method – such as compressed sparse row (CSR) or compressed row storage (CRS) or Yale format may be implemented to generate compressed SH coefficient data to be encoded in a coded bitstream as described herein.

[0150] Additionally, optionally or alternatively, in some operational scenarios, HDF5 format or the like may be used as a relatively high dimension data storage and compression format. Using a selected chunk size and a selected compression method, SH coefficient data in a 3D SH cube or in separate 2D SH planes may be encoded into compressed SH coefficient data to be encoded in a coded bitstream as described herein.

[0151] Example operations with HDF5 format are described in “High Level Introduction to HDF5,” available at https: / / docs.hdfgroup.org / archive / support / HDF5 / Tutor / HDF5Intro.pdf (accessed on January 19, 2024), which is incorporated by reference herein in its entirety.

[0152] In some operational scenarios, a spatial frequency (e.g., optimal, optimized, etc.) quantizer may be used in SH coefficient compression. For example, an (e.g., optimal, optimized, etc.) 1D Lloyd-Max non-linear quantizer can deployed to quantize an SH 2D coefficients plane. A histogram of SH coefficient magnitudes may be built first and then the quantizer can be designed or performed to minimize errors such as measured with mean- squared-errors (MSEs). The quantizer outputs or generates decision boundaries and reconstruction level(s). The reconstruction level(s) may be signaled in image or LSLF metadata in a coded bitstream as described herein. Additionally, optionally or alternatively, an (e.g., optimal, optimized, etc.) vector quantizer can be used to quantize an SH 3D coefficients cube. The codebook for each vector used by the vector quantizer may be signaled in the image or LSLF metadata in the coded bitstream. 14. COEFFICIENTS REGENERATION

[0153] Denote SH coefficients on the p-th 2D plane as "#, where p is an integer from 0 to (P-1), where P represents the total number of 2D planes, respectively corresponding from SH0 to SH(P-1). Denote an element or instance of "#(or SH coefficients on the p-th 2Dplane) such as a specific SH coefficient at a specific 2D position with coordinates (x, y) as!#$% .

[0154] In the previous SuRF use cases or examples (in which the 0th, first, and second orders of SH basis functions are used to represent view-direction-dependent intensity and / or color information of a point on a surface layer of an LSLF set), (P=9), P equals 8, and hence p is between

[0008] .

[0155] It may be challenging for some existing image codecs to directly encode or decode SH coefficients with relatively high compression efficiency while maintaining or achieving a relatively high LSLF or image reconstruction quality. This is because the SH coefficients do not resemble nature images (except p=0). As a result, spatial / temporal correlations among the SH coefficients may be significantly different from those among image data values in nature images.

[0156] Many available video / image codecs may be relatively fine-tuned for processing or operating with nature image. Under techniques as described herein, to make SH coefficient compression more efficient for these codecs, input or original SH coefficients in an LSLF set as described herein – or the P input or original 2D planes on which input or original SH coefficients are located – may be first synthesized or resynthesized into synthesized or resynthesized P – e.g., more “nature-image like” – 2D planes from those input or original SH coefficients. In other words, instead of carrying the input or original SH coefficients, the synthesized or resynthesized P 2D planes carry or host synthesized or resynthesized SH coefficients that have spatial / temporal correlations similar to those in natural images (e.g., captured with cameras, RGB valued images, YUV valued images, etc.).

[0157] The P synthesized or resynthesized 2D planes or synthesized or resynthesized SH coefficients therewith may be encoded into (by an upstream device) or decoded from (by a downstream recipient device) a coded bitstream using available video / image codecs to achieve a relatively high compression efficiency. More specifically, on the decoder side, the downstream device can re-solve or reconstruct SH coefficients relatively closely approximating the input or original SH coefficients using the P synthesized or resynthesized 2D planes – or decoded images specified by the synthesized or resynthesized SH coefficients decoded from the bitstream.

[0158] An example procedure or process flow to synthesize or resynthesize the input or original P 2D planes may be implemented or performed as follows.

[0159] For each p, where p = 0, …, P-1, a vector denoted as &#may be constructed to contain all P SH basis functions from all orders using P selected (or p dependent) anglesdenoted as (^#, ^#) as follows:&# = [^' # # ^ # #' (^ , ^ ) … ^^ (^ , ^ )] (3)

[0160] For each pixel location (x, y), P SH coefficients from all different SH planes {!$#% } may be collected into a column vector as follows: !$'%)$% = *(4)

[0161] %at thepixel location (x, y) can be computed as follows: ℎ# #$% = & )$% (5)

[0162] All these image data or pixel values at all pixel locations (x, y) for the p-th plane may be collected into a matrix or vector denoted as 0#with all elements ℎ$#% in expression (5). These {0#} or image data or pixel values can be encoded by the upstream device intothe coded bitstream using one or more available video / image codecs. The set of angles{(^#, ^#)} can be transmitted as image metadata along with the image data or pixel values inthe coded bitstream.

[0163] On the decoder side, &#can be reconstructed by the downstream recipient deviceof the bitstream using the set of angles {(^#, ^#)} determined from the image metadatacarried in the bitstream. The decompressed 2D plane may be represented or denoted as 03#with elements ℎ4$#% .

[0164] On the encoder side, optimized values for the vectors &#may be determined or generated as follows.

[0165] The vectors &#for all p from 0 to (P-1) may be collected into an overall vector as follows: &'(6)

[0166] {x, y} across all P planes – synthesized pixel (x, y) may be arranged or collected into a vector as follows:ℎ4$'%83$% = * / (7)

[0167] as !̂$#% .All !̂$'%:;$% = *(8)

[0168] can beobtained location, as follows: :;<#= > -. >$% = (5 5) ?5 83$%@ (9)

[0169] depends on or may bedetermined by search for the best reconstructed rendered images via the optimized values for the SH coefficients in expression (9).

[0170] In many operational scenarios, the SuRF algorithm can be applied or used to build from a collection of U captured images from different camera poses {AB}. An example procedure or process flow to determine or generate a relatively high-quality set of camera poses is illustrated in TABLE 3 below.

[0171] As shown in TABLE 3, the procedure comprises STEP 1 (Initialization) in which the k-mean clustering algorithm may be applied on the different camera poses {CB} to obtain P clusters. In some operational scenarios, the centroid of each p-th cluster can be used toderive a corresponding pair of angles (^#, ^#) for the p-th cluster.

[0172] The procedure further comprises STEP 2 (Fine Tuning) in which an iterative greedy algorithm can be applied to adjust each angle or each pair of angles one by one to determine whether any improvement is to be made. TABLE 3 k=0; / / iteration index while convergence = =0convergence = convergence + 1 else () KLM ( ) KLM ( ) KLM ( ) KLM ( ) KLM ( KLMupdate ?^ E ,# ^ E ,# @ = ?^ E-. ,# ^ E-. ,# @ + ?∆^ E ,# ∆^ E),# @render visual images related to a SuRF format such as LSLF sets. It should be noted, however, that some or all of these techniques are not necessarily limited to or constrained to a SuRF format such as LSLF sets. In other operational scenarios, these techniques may be extended to other non-SuRF formats or other datasets that use – e.g., all orders, or (non 0th order) higher-order – spherical harmonics. 15. EXAMPLE PROCESS FLOWS

[0174] FIG.4A illustrates an example process flow according to an embodiment. In some embodiments, one or more computing devices or components (e.g., one or more video codecs, an encoding device / module, a transcoding device / module, a volumetric or immersive video server system, etc.) may perform this process flow. In block 402, an image processing system receives an input light field data set, wherein the input light field data set includes one or more surface layers. Each surface layer in the one or more surface layers of the input light field data set includes depth data, alpha map data, multiple sets of spherical harmonics (SH) coefficients of multiple different combinations of SH basis function order and SH basis function index, etc.

[0175] In block 404, the system converts the depth data, alpha map data, and the multiple sets of SH coefficients to a depth image, an alpha map image, and multiple SH coefficient images.

[0176] In block 406, the system performs one or more encoder-side operations on at least one of the multiple SH coefficient images to generate an alternative SH coefficient data representation different from an input SH coefficient data representation in which the at least one of the multiple SH coefficient images is represented.

[0177] In block 408, the system encodes the alternative SH coefficient data representation, in place of the at least one of the multiple SH coefficient images, into a coded bitstream along with the depth image and the alpha image. The coded bitstream causes a recipient device of the coded bitstream to generate a reconstructed light field data set that approximates the input light field data set.

[0178] In an embodiment, the multiple SH coefficient images includes at least (a) a first SH coefficient image consisting of SH coefficients of a 0th order and (b) a second SH coefficient image consisting of SH coefficients of a non-0th order.

[0179] In an embodiment, an encoder-side operation performed on the first SH coefficient image is different from an encoder-side operation performed on the second SH coefficient image. Specifically, the encoder-side operation for the second SH coefficient image comprises a data pruning operation.

[0180] In an embodiment, the multiple SH coefficient images includes a plurality of SH coefficient images consisting of SH coefficients of a plurality of non-0th orders; the plurality of plurality of SH coefficient images is respective indexed by a plurality of distinct index values.

[0181] In an embodiment, pixel locations of non-zero SH coefficients are represented as floating point values in two coordinates X and Y of a floating point plane identified by an index values of a SH basis function; the plurality of distinct index values used to index the plurality of plurality of SH coefficient images is combined with the pixel locations of the non-zero SH coefficients to form a geometry of a point cloud in a three dimensional (3D) space comprising a plurality of floating point planes identified by respective index values of SH basis functions; the one or more encoder-side operations includes geometric point cloud compression operations performed on the geometry of the point cloud along with respective attributes represented by the non-zero SH coefficients.

[0182] In an embodiment, the one or more encoder-side operations includes selecting a plurality of sets of azimuth and elevation angles for the plurality of plurality of SH coefficient images specifically to minimize reconstruction quality errors; the plurality of sets of azimuth and elevation angles is combined with the plurality of SH coefficient images to form a synthesized SH image; the synthesized image is encoded into the coded bitstream in place of the plurality of SH coefficient images.

[0183] In an embodiment, the one or more encoder-side operations includes packing and encoding quad-tree sub-blocks containing at least one non-zero SH coefficient into the coded bitstream.

[0184] In an embodiment, the one or more encoder-side operations includes packing the SH coefficient images into one or more atlases and encoding quad-tree sub-blocks, of the one or more atlases, each of which contains at least one non-zero SH coefficient into the coded bitstream.

[0185] In an embodiment, the one or more encoder-side operations includes artificialneural network based geometry coding operations that use one or more trained AI models to predict locations of non-zero SH coefficients.

[0186] In an embodiment, the one or more encoder-side operations includes mesh based geometry coding operations that generates a base mesh geometry sub-bitstream in the coded bitstream and a displacement geometry refinement sub-bitstream in the coded bitstream.

[0187] In an embodiment, SH coefficient data encoded into the coded bitstream represents residual data generated with one or more prediction operations.

[0188] In an embodiment, the one or more prediction operations includes using a Lorenzo predictor to generate the residual data.

[0189] In an embodiment, at least one of the encoder-side operations is performed in a transform domain other than a spatial domain in which locations of the SH coefficients are represented.

[0190] In an embodiment, the process flow as described herein is performed by an upstream encoding device implemented at least in part by one or more video codecs relating to one of: Moving Picture Experts Group (MPEG), Advanced Video Coding (AVC), High- Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Essential Video Coding (EVC), Point Cloud Compression (PCC), Video-based Point Cloud Compression (V-PCC), Visual Volumetric Video-based Coding (V3C), or the like.

[0191] In an embodiment, the process flow as described herein is performed by an upstream encoding device implemented at least in part by one or more video codecs relating to one of: Moving Picture Experts Group (MPEG), Advanced Video Coding (AVC), High- Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Essential Video Coding (EVC), Point Cloud Compression (PCC), Video-based Point Cloud Compression (V-PCC), Visual Volumetric Video-based Coding (V3C), or the like.

[0192] FIG.4B illustrates an example process flow according to an embodiment. In some embodiments, one or more computing devices or components (e.g., one or more video codecs, a coding device / module, a transcoding device / module, a decoding device / module, a volumetric or immersive video client device or system, etc.) may perform this process flow. In block 452, an image processing system decodes an alternative spherical harmonics (SH) coefficient data representation, different from an original SH coefficient data representation used to derive the alternative SH coefficient data representation, from a coded bitstream along with a depth image and an alpha image. The alterative SH coefficient data representation is generated by an upstream device in place of at least one of multiple SH coefficient images in an input light field data set.

[0193] In block 454, based at least in part on the depth image, the alpha map image, and the alternative SH coefficient data representation, the system generates a reconstructed light field data set that approximates the input light field data set.

[0194] In block 456, the system uses the reconstructed light field data set to generate a display image from the reconstructed light field data set.

[0195] In block 458, the system renders the display image on an image display.

[0196] In an embodiment, a computing device such as a display device, a mobile device, a set-top box, a multimedia device, etc., is configured to perform any of the foregoing methods. In an embodiment, an apparatus comprises a processor and is configured to perform any of the foregoing methods. In an embodiment, a non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors cause performance of any of the foregoing methods.

[0197] In an embodiment, a computing device comprising one or more processors and one or more storage media storing a set of instructions which, when executed by the one or more processors, cause performance of any of the foregoing methods.

[0198] Note that, although separate embodiments are discussed herein, any combination of embodiments and / or partial embodiments discussed herein may be combined to form further embodiments. 16. IMPLEMENTATION MECHANISMS – HARDWARE OVERVIEW

[0199] Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or apparatus that includes one or more of such systems, devices or components. The computer and / or IC may perform, control, or execute instructions relating to the adaptive perceptual quantization of images with enhanced dynamic range, such as those described herein. The computer and / or IC may compute any of a variety of parameters or values that relate to the adaptive perceptual quantization processes described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.

[0200] Certain implementations of the inventio comprise computer processors which execute software instructions which cause the processors to perform a method of the disclosure. For example, one or more processors in a display, an encoder, a set top box, a transcoder or the like may implement methods related to adaptive perceptual quantization ofHDR images as described above by executing software instructions in a program memory accessible to the processors. Embodiments of the invention may also be provided in the form of a program product. The program product may comprise any non-transitory medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of an embodiment of the invention. Program products according to embodiments of the invention may be in any of a wide variety of forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0201] Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.

[0202] According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and / or program logic to implement the techniques.

[0203] For example, FIG.5 is a block diagram that illustrates a computer system 500 upon which an embodiment of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled with bus 502 for processing information. Hardwareprocessor 504 may be, for example, a general purpose microprocessor.

[0204] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in non-transitory storage media accessible to processor 504, render computer system 500 into a special-purpose machine that is customized to perform the operations specified in the instructions.

[0205] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic disk or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0206] Computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to bus 502 for communicating information and command selections to processor 504. Another type of user input device is cursor control 516, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 504 and for controlling cursor movement on display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.

[0207] Computer system 500 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and / or program logic which in combination with the computer system causes or programs computer system 500 to be a special-purpose machine. According to one embodiment, the techniques as described herein are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0208] The term “storage media” as used herein refers to any non-transitory media that store data and / or instructions that cause a machine to operation in a specific fashion. Such storage media may comprise non-volatile media and / or volatile media. Non-volatile media includes, for example, optical or magnetic disks, such as storage device 510. Volatile mediaincludes dynamic memory, such as main memory 506. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0209] Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.

[0210] Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 500 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 502. Bus 502 carries the data to main memory 506, from which processor 504 retrieves and executes the instructions. The instructions received by main memory 506 may optionally be stored on storage device 510 either before or after execution by processor 504.

[0211] Computer system 500 also includes a communication interface 518 coupled to bus 502. Communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, communication interface 518 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 518 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0212] Network link 520 typically provides data communication through one or more networks to other data devices. For example, network link 520 may provide a connectionthrough local network 522 to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526. ISP 526 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 528. Local network 522 and Internet 528 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 520 and through communication interface 518, which carry the digital data to and from computer system 500, are example forms of transmission media.

[0213] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet example, a server 530 might transmit a requested code for an application program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0214] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution. 17. EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

[0215] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is claimed embodiments of the invention, and is intended by the applicants to be claimed embodiments of the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. Enumerated Exemplary Embodiments

[0216] The invention may be embodied in any of the forms described herein, including, but not limited to the following Enumerated Example Embodiments (EEEs) which describe structure, features, and functionality of some portions of embodiments of the present invention.

[0217] EEE1. A method, comprising: receiving an input light field data set, wherein the input light field data set includesone or more surface layers, wherein each surface layer in the one or more surface layers of the input light field data set includes depth data, alpha map data, and multiple sets of spherical harmonics (SH) coefficients of multiple different combinations of SH basis function order and SH basis function index; converting the depth data, alpha map data, and the multiple sets of SH coefficients to a depth image, an alpha map image, and multiple SH coefficient images; performing one or more encoder-side operations on at least one of the multiple SH coefficient images to generate an alternative SH coefficient data representation different from an input SH coefficient data representation in which the at least one of the multiple SH coefficient images is represented; encoding the alternative SH coefficient data representation, in place of the at least one of the multiple SH coefficient images, into a coded bitstream along with the depth image and the alpha image, wherein the coded bitstream causes a recipient device of the coded bitstream to generate a reconstructed light field data set that approximates the input light field data set.

[0218] EEE2. The method as recited in EEE1, wherein each surface layer in the light field data set are partitioned into non-overlapping patches; wherein a subset of the non- overlapping patches with nonzero coefficients are identified as significant patches; wherein an atlas image is formed by the significant patches; wherein each of the significant patches in each said surface layer in the light field data set is encoded in the coded bitstream by way of encoding the atlas image into the coded bitstream.

[0219] EEE3. The method as recited in EEE1 or EEE2, wherein each set of SH coefficients is collected into a respective SH image partitioned into coding tree blocks (CTBs); wherein each CTB in the CTBs is represented in a quadtree data structure with the CTB as a root node and with coding blocks (CBs) partitioned from the CTB as non-root nodes; wherein each CB in the CBs is partitioned into coding sub-blocks; wherein each coding sub-block, in the coding sub-blocks, with non-zero SH coefficients is encoded in the coded bitstream.

[0220] EEE4. The method as recited in any of EEE1-EEE3, wherein the multiple SH coefficient images includes at least (a) a first SH coefficient image consisting of SH coefficients of a 0th order and (b) a second SH coefficient image consisting of SH coefficients of a non-0th order.

[0221] EEE5. The method as recited in any of EEE1-EEE4, wherein the multiple SH coefficient images includes a plurality of SH coefficient images consisting of SH coefficients of a plurality of non-0th orders; wherein the plurality of plurality of SH coefficient images isrespective indexed by a plurality of distinct index values.

[0222] EEE6. The method as recited in EEE5, wherein pixel locations of non-zero SH coefficients are represented as floating point values in two coordinates X and Y of a floating point plane identified by an index values of a SH basis function; wherein the plurality of distinct index values used to index the plurality of plurality of SH coefficient images is combined with the pixel locations of the non-zero SH coefficients to form a geometry of a point cloud in a three dimensional (3D) space comprising a plurality of floating point planes identified by respective index values of SH basis functions; wherein the one or more encoder-side operations includes geometric point cloud compression operations performed on the geometry of the point cloud along with respective attributes represented by the non-zero SH coefficients.

[0223] EEE7. The method as recited in EEE5, wherein the one or more encoder-side operations includes selecting a plurality of sets of azimuth and elevation angles for the plurality of plurality of SH coefficient images specifically to minimize reconstruction quality errors; wherein the plurality of sets of azimuth and elevation angles is combined with the plurality of SH coefficient images to form a synthesized SH image comprising regenerated SH coefficients; wherein the synthesized image with the regenerated SH coefficients is encoded into the coded bitstream in place of the plurality of SH coefficient images.

[0224] EEE8. The method as recited in EEE5, wherein the one or more encoder-side operations includes packing and encoding quad-tree sub-blocks containing at least one non- zero SH coefficient into the coded bitstream.

[0225] EEE9. The method as recited in EEE5, wherein the one or more encoder-side operations includes packing the SH coefficient images into one or more atlases and encoding quad-tree sub-blocks, of the one or more atlases, each of which contains at least one non-zero SH coefficient into the coded bitstream.

[0226] EEE10. The method as recited in EEE5, wherein the one or more encoder-side operations includes artificial neural network based geometry coding operations that use one or more trained AI models to predict locations of non-zero SH coefficients.

[0227] EEE11. The method as recited in EEE5, wherein the one or more encoder-side operations includes mesh based geometry coding operations that generates a base mesh geometry sub-bitstream in the coded bitstream and a displacement geometry refinement sub- bitstream in the coded bitstream.

[0228] EEE12. The method as recited in any of EEE1-EEE11, wherein SH coefficient data encoded into the coded bitstream represents residual data generated with one or moreprediction operations.

[0229] EEE13. The method as recited in EEE12, wherein the one or more prediction operations includes using one of a three-dimensional (3D) predictor or a Lorenzo predictor to generate the residual data.

[0230] EEE14. The method as recited in any of EEE1-EEE13, wherein SH coefficient data is represented in a plurality of quantized SH planes containing relatively sparse non-zero SH coefficients; wherein a sparse matrix compression method is applied to the plurality of quantized SH planes to encode the SH coefficient data into the coded bitstream; wherein the sparse matrix compression method is one of: a compressed sparse row (CSR) based method, a compressed row storage (CRS) based method, or a Yale format based method.

[0231] EEE15. The method as recited in any of EEE1-EEE14, wherein at least one of the encoder-side operations is performed in a transform domain other than a spatial domain in which locations of the SH coefficients are represented.

[0232] EEE16. The method as recited in any of EEE1-EEE15, wherein the method is performed by an upstream encoding device implemented at least in part by one or more video codecs relating to one of: Moving Picture Experts Group (MPEG), Advanced Video Coding (AVC), High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Essential Video Coding (EVC), Point Cloud Compression (PCC), Video- based Point Cloud Compression (V-PCC), or Visual Volumetric Video-based Coding (V3C).

[0233] EEE17. The method as recited in any of EEE1-EEE16, wherein the method is performed by an upstream encoding device implemented at least in part by one or more video codecs relating to one of: Moving Picture Experts Group (MPEG), Advanced Video Coding (AVC), High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Essential Video Coding (EVC), Point Cloud Compression (PCC), Video- based Point Cloud Compression (V-PCC), or Visual Volumetric Video-based Coding (V3C).

[0234] EEE18. A method, comprising: decoding an alternative spherical harmonics (SH) coefficient data representation, different from an original SH coefficient data representation used to derive the alternative SH coefficient data representation, from a coded bitstream along with a depth image and an alpha image, wherein the alterative SH coefficient data representation is generated by an upstream device in place of at least one of multiple SH coefficient images in an input light field data set; based at least in part on the depth image, the alpha map image, and the alternative SH coefficient data representation, generating a reconstructed light field data set thatapproximates the input light field data set; using the reconstructed light field data set to generate a display image from the reconstructed light field data set; rendering the display image on an image display.

[0235] EEE19. A method, comprising: receiving an input light field data set, wherein the input light field data set includes one or more surface layers, wherein each surface layer in the one or more surface layers of the input light field data set includes depth data, alpha map data, and multiple sets of non- spherical harmonics (non-SH) coefficients of multiple different combinations of SH basis function order and non-SH basis function index; converting the depth data, alpha map data, and the multiple sets of non-SH coefficients to a depth image, an alpha map image, and multiple SH coefficient images; performing one or more encoder-side operations on at least one of the multiple non- SH coefficient images to generate an alternative non-SH coefficient data representation different from an input non-SH coefficient data representation in which the at least one of the multiple non-SH coefficient images is represented; encoding the alternative non-SH coefficient data representation, in place of the at least one of the multiple non-SH coefficient images, into a coded bitstream along with the depth image and the alpha image, wherein the coded bitstream causes a recipient device of the coded bitstream to generate a reconstructed light field data set that approximates the input light field data set.

[0236] EEE20. A method, comprising: decoding an alternative non-spherical harmonics (non-SH) coefficient data representation, different from an original non-SH coefficient data representation used to derive the alternative non-SH coefficient data representation, from a coded bitstream along with a depth image and an alpha image, wherein the alterative non-SH coefficient data representation is generated by an upstream device in place of at least one of multiple non-SH coefficient images in an input light field data set; based at least in part on the depth image, the alpha map image, and the alternative non-SH coefficient data representation, generating a reconstructed light field data set that approximates the input light field data set; using the reconstructed light field data set to generate a display image from the reconstructed light field data set; rendering the display image on an image display.

[0237] EEE21. An apparatus performing any of the methods as recited in EEE1- EEE20.

[0238] EEE22. A non-transitory computer readable medium, storing software instructions, which when executed by one or more processors cause performance of the steps of any of the methods as recited in EEE1-EEE20.

[0239] EEE23. A computing device comprising one or more processors and one or more storage media, storing a set of instructions, which when executed by one or more processors cause performance of the method recited in any of EEE1-EEE20.

Claims

CLAIMS 1. A method, comprising: receiving an input light field data set, wherein the input light field data set includes one or more surface layers, wherein each surface layer in the one or more surface layers of the input light field data set includes depth data, alpha map data, and multiple sets of spherical harmonics (SH) coefficients of multiple different combinations of SH basis function order and SH basis function index; converting the depth data, alpha map data, and the multiple sets of SH coefficients to a depth image, an alpha map image, and multiple SH coefficient images; performing one or more encoder-side operations on at least one of the multiple SH coefficient images to generate an alternative SH coefficient data representation different from an input SH coefficient data representation in which the at least one of the multiple SH coefficient images is represented; encoding the alternative SH coefficient data representation, in place of the at least one of the multiple SH coefficient images, into a coded bitstream along with the depth image and the alpha image, wherein the coded bitstream causes a recipient device of the coded bitstream to generate a reconstructed light field data set that approximates the input light field data set.

2. The method as recited in claim 1, wherein each surface layer in the light field data set are partitioned into non-overlapping patches; wherein a subset of the non-overlapping patches with nonzero coefficients are identified as significant patches; wherein an atlas image is formed by the significant patches; wherein each of the significant patches in each said surface layer in the light field data set is encoded in the coded bitstream by way of encoding the atlas image into the coded bitstream.

3. The method as recited in claim 1 or 2, wherein each set of SH coefficients is collected into a respective SH image partitioned into coding tree blocks (CTBs); wherein eachCTB in the CTBs is represented in a quadtree data structure with the CTB as a root node and with coding blocks (CBs) partitioned from the CTB as non-root nodes; wherein each CB in the CBs is partitioned into coding sub-blocks; wherein each coding sub-block, in the coding sub-blocks, with non-zero SH coefficients is encoded in the coded bitstream.

4. The method as recited in any one of claims 1-3, wherein the multiple SH coefficient images includes at least (a) a first SH coefficient image consisting of SH coefficients of a 0th order and (b) a second SH coefficient image consisting of SH coefficients of a non-0th order.

5. The method of claim 4, wherein an encoder-side operation performed on the first SH coefficient image is different from an encoder-side operation performed on the second SH coefficient image.

6. The method of claim 5, wherein the encoder-side operation for the second SH coefficient image comprises a data pruning operation.

7. The method as recited in any one of claims 1-6, wherein the multiple SH coefficient images includes a plurality of SH coefficient images consisting of SH coefficients of a plurality of non-0th orders; wherein the plurality of plurality of SH coefficient images is respective indexed by a plurality of distinct index values.

8. The method as recited in claim 7, wherein pixel locations of non-zero SH coefficients are represented as floating point values in two coordinates X and Y of a floating point plane identified by an index values of a SH basis function; wherein the plurality of distinct index values used to index the plurality of plurality of SH coefficient images is combined with the pixel locations of the non-zero SH coefficients to form a geometry of a point cloud in a three dimensional (3D) space comprising a plurality of floating point planes identified by respective index values of SH basis functions;wherein the one or more encoder-side operations includes geometric point cloud compression operations performed on the geometry of the point cloud along with respective attributes represented by the non-zero SH coefficients.

9. The method as recited in claim 7, wherein the one or more encoder-side operations includes selecting a plurality of sets of azimuth and elevation angles for the plurality of plurality of SH coefficient images specifically to minimize reconstruction quality errors; wherein the plurality of sets of azimuth and elevation angles is combined with the plurality of SH coefficient images to form a synthesized SH image comprising regenerated SH coefficients; wherein the synthesized image with the regenerated SH coefficients is encoded into the coded bitstream in place of the plurality of SH coefficient images.

10. The method as recited in claim 7, wherein the one or more encoder-side operations includes packing and encoding quad-tree sub-blocks containing at least one non-zero SH coefficient into the coded bitstream.

11. The method as recited in claim 7, wherein the one or more encoder-side operations includes packing the SH coefficient images into one or more atlases and encoding quad-tree sub-blocks, of the one or more atlases, each of which contains at least one non-zero SH coefficient into the coded bitstream.

12. The method as recited in claim 7, wherein the one or more encoder-side operations includes artificial neural network based geometry coding operations that use one or more trained AI models to predict locations of non-zero SH coefficients.

13. The method as recited in claim 7, wherein the one or more encoder-side operations includes mesh based geometry coding operations that generates a base mesh geometry sub-bitstream in the coded bitstream and a displacement geometry refinement sub- bitstream in the coded bitstream.

14. The method as recited in any one of claims 1-13, wherein SH coefficient data encoded into the coded bitstream represents residual data generated with one or more prediction operations.

15. The method as recited in claim 14, wherein the one or more prediction operations includes using one of a three-dimensional (3D) predictor or a Lorenzo predictor to generate the residual data.

16. The method as recited in any one of claims 1-15, wherein SH coefficient data is represented in a plurality of quantized SH planes containing relatively sparse non-zero SH coefficients; wherein a sparse matrix compression method is applied to the plurality of quantized SH planes to encode the SH coefficient data into the coded bitstream; wherein the sparse matrix compression method is one of: a compressed sparse row (CSR) based method, a compressed row storage (CRS) based method, or a Yale format based method.

17. The method as recited in any one of claims 1-16, wherein at least one of the encoder- side operations is performed in a transform domain other than a spatial domain in which locations of the SH coefficients are represented.

18. The method as recited in any one of claims 1-17, wherein the method is performed by an upstream encoding device implemented at least in part by one or more video codecs relating to one of: Moving Picture Experts Group (MPEG), Advanced Video Coding (AVC), High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), AOMedia Video 1 (AV1), Essential Video Coding (EVC), Point Cloud Compression (PCC), Video-based Point Cloud Compression (V-PCC), or Visual Volumetric Video-based Coding (V3C).

19. A method, comprising: decoding an alternative spherical harmonics (SH) coefficient data representation, different from an original SH coefficient data representation used to derive the alternative SH coefficient data representation, from a coded bitstream along with a depth image and an alpha image, wherein the alterative SH coefficient data representation is generated by an upstream device in place of at least one of multiple SH coefficient images in an input light field data set; based at least in part on the depth image, the alpha map image, and the alternative SH coefficient data representation, generating a reconstructed light field data set that approximates the input light field data set; using the reconstructed light field data set to generate a display image from the reconstructed light field data set; rendering the display image on an image display.

20. An apparatus configured to perform the method of any one of claims 1-19.

21. A non-transitory computer readable medium, storing software instructions, which when executed by one or more processors cause performance of the steps of the method of any one of claims 1-19.

Images