Image compression using tensor product B-spline representation

Abstract

A method and apparatus for image compression / decompression using a tensor product B-spline (TPB) representation. According to one example embodiment, the image compression method includes generating a plurality of TPB models representing an image, including a TPB model for approximating a spatial reshaping function and a plurality of patch-wise TPB models for estimating an image signal. In some examples, the plurality of patch-wise TPB models include a plurality of luma channel TPB models and a plurality of chroma channel TPB models. The spatial reshaping function is configured to shift a non-uniform distribution of local content complexity within the image to a more uniform distribution. The method also includes generating a metadata stream carrying metadata representing sets of coefficients of the various TPB models. At least a portion of the metadata is generated by quantizing corresponding coefficients to a fewer number of bits and applying arithmetic coding to the quantized coefficients.

Description

[Technical Field]

[0001] 1. CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 484,329, filed February 10, 2023, and European Patent Application No. 23167592.7, filed April 12, 2023, both of which are incorporated herein by reference.

[0002] 2.Technical Field Various example embodiments relate to image manipulation, and more particularly, but not exclusively, to image compression. [Background technology]

[0003] 3.Background technology Image compression is a process applied to image files to reduce their file size in bytes without degrading image quality below a selected quality threshold. Reducing file size allows more images to be stored in a given amount of memory. Compressed images also require less bandwidth to transmit over communication channels, which can reduce network congestion and speed up content delivery. Summary of the Invention

[0004] Various embodiments of methods and apparatuses for image compression / decompression using tensor product B-spline (TPB) representation are disclosed herein. According to one example embodiment, the image compression method includes generating multiple TPB models representing an image, including a TPB model for approximating a spatial reshaping function and multiple patch-wise TPB models for estimating an image signal. In some examples, the multiple patch-wise TPB models include multiple luma (luminance) channel TPB models and multiple chroma (color difference) channel TPB models. The spatial reshaping function is configured to shift a non-uniform distribution of local content complexity within the image to a more uniform distribution. The method also includes generating a metadata stream carrying metadata representing sets of coefficients of the various TPB models. At least a portion of the metadata is generated by quantizing corresponding coefficients to fewer bits and applying arithmetic coding to the quantized coefficients.

[0005] According to one example embodiment, there is provided a method for compressing an image, the method including: generating, using an electronic encoder, a first TPB model representing an image, the first TPB model approximating a spatial reshaping function configured to map uniformly sampled locations in the image to non-uniformly spaced locations; generating, using the electronic encoder, a plurality of second TPB models, each representing a respective patch of the image and configured to estimate image signals in each patch as a function of reshaped coordinates, the reshaped coordinates being determined using the spatial reshaping function; and generating, using the electronic encoder, a metadata stream including first metadata representing a set of coefficients of the first TPB model and second metadata representing a set of coefficients of the plurality of second TPB models.

[0006] According to another example embodiment, an apparatus for image compression is provided, the apparatus comprising: at least one processor; and at least one memory containing program code configured, using the at least one processor, to cause the apparatus to at least: generate a first TPB model representing an image, the first TPB model approximating a spatial reshaping function configured to map uniformly sampled locations in the image to non-uniformly spaced locations; generate a plurality of second TPB models, each representing a respective patch of the image and configured to estimate image signals in each patch as a function of reshaped coordinates, the reshaped coordinates being determined using the spatial reshaping function; and generate a metadata stream including first metadata representing a set of coefficients of the first TPB model and second metadata representing the set of coefficients of the plurality of second TPB models.

[0007] According to yet another example embodiment, a non-transitory computer-readable medium is provided that stores instructions that, when executed by an electronic processor, cause the electronic processor to perform operations including a method for image compression, including: generating, with an electronic encoder, a first TPB model representing an image, the first TPB model approximating a spatial reshaping function configured to map uniformly sampled locations in the image to non-uniformly spaced locations; generating, with the electronic encoder, a plurality of second TPB models, each representing a respective patch of the image and configured to estimate image signals in each patch as a function of reshaped coordinates, the reshaped coordinates being determined using the spatial reshaping function; and generating, with the electronic encoder, a metadata stream including first metadata representing a set of coefficients of the first TPB model and second metadata representing a set of coefficients of the plurality of second TPB models.

[0008] According to yet another example embodiment, there is provided a method of image decompression that includes receiving, with an electronic decoder, a metadata stream including first metadata representing a set of coefficients of a first TPB model and second metadata representing a set of coefficients of a plurality of second TPB models, the first TPB model approximating a spatial reshaping function configured to map uniformly sampled locations in a source image to non-uniformly spaced locations, each of the second TPB models representing a respective patch of the source image and configured to estimate image signals in each patch as a function of reshaped coordinates, the reshaped coordinates being determined based on the spatial reshaping function, generating, with the electronic decoder, a plurality of reconstructed image patches based on the first TPB model and further based on the plurality of second TPB models, where the first TPB model is reconstructed using the first metadata and the plurality of second TPB models are reconstructed using the second metadata, and constructing, with the electronic decoder, an estimated image by assembling the plurality of reconstructed image patches in an image frame.

[0009] According to yet another example embodiment, an apparatus for image decompression is provided, the apparatus comprising: at least one processor; and at least one memory containing program code, the at least one memory and the program code being configured to cause, using the at least one processor, the apparatus to receive a metadata stream containing at least first metadata representing a set of coefficients of a first TPB model and second metadata representing sets of coefficients of a plurality of second TPB models, the first TPB model approximating a spatial reshaping function configured to map uniformly sampled locations in a source image to non-uniformly spaced locations, each of the second TPB models representing a respective patch of the source image and each of the second TPB models representing a respective patch of the source image. as a function of reshaped coordinates, the reshaped coordinates being determined using a spatial reshaping function; generating a plurality of reconstructed image patches based on a first TPB model and further based on a plurality of second TPB models, the first TPB model being reconstructed using the first metadata and the plurality of second TPB models being reconstructed using the second metadata; and constructing an estimated image by assembling the plurality of reconstructed image patches in an image frame.

[0010] According to yet another example embodiment, a non-transitory computer-readable medium is provided that stores instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising an image decompression method. A method for image decompression includes receiving, with an electronic decoder, a metadata stream including first metadata representing a set of coefficients of a first TPB model and second metadata representing sets of coefficients of a plurality of second TPB models, where the first TPB model approximates a spatial reshaping function configured to map uniformly sampled locations in a source image to non-uniformly spaced locations, and where each of the second TPB models represents a respective patch of the source image and is configured to estimate an image signal in each patch as a function of reshaped coordinates, the reshaped coordinates being determined based on the spatial reshaping function; generating, with the electronic decoder, a plurality of reconstructed image patches based on the first TPB model and further based on a plurality of second TPB models, where the first TPB model is reconstructed using the first metadata and the plurality of second TPB models are reconstructed using the second metadata; and constructing, with the electronic decoder, an estimated image by assembling the plurality of reconstructed image patches in an image frame. [Brief explanation of the drawings]

[0011] Other aspects, features, and advantages of the various disclosed embodiments will become more fully apparent, by way of example, from the following detailed description and the accompanying drawings. [Figure 1] FIG. 1 is a block diagram illustrating an example of a process for a video / image delivery pipeline. [Figure 2] FIG. 2 is a block diagram illustrating an encoder usable in the distribution pipeline of FIG. 1 according to one embodiment. [Figure 3] FIG. 2 is a block diagram illustrating a decoder usable in the distribution pipeline of FIG. 1 according to one embodiment. [Figure 4A]FIG. 3 graphically illustrates a spatial reshaping function calculated in the encoder of FIG. 2 according to an example. [Figure 4B] FIG. 3 graphically illustrates a spatial reshaping function calculated in the encoder of FIG. 2 according to an example. [Figure 5] FIG. 3 is a block diagram illustrating the compression operations performed in the encoder of FIG. 2 according to one embodiment. [Figure 6] 10 shows a bar chart graphically illustrating the distribution of uniformly quantized TPB coefficient values ​​according to an example. [Figure 7A] 4 is a block diagram illustrating an example format of a metadata stream transmitted from the encoder of FIG. 2 to the decoder of FIG. 3 according to one embodiment. [Figure 7B] 4 is a block diagram illustrating an example format of a metadata stream transmitted from the encoder of FIG. 2 to the decoder of FIG. 3 according to one embodiment. [Figure 8A] 1 is a graph illustrating a set of B-spline basis functions of orders 0 to 3 for a uniform distribution of knot points according to an example. [Figure 8B] 1 is a graph illustrating a set of B-spline basis functions of orders 0 to 3 for a uniform distribution of knot points according to an example. [Figure 8C] 1 is a graph illustrating a set of B-spline basis functions of orders 0 to 3 for a uniform distribution of knot points according to an example. [Figure 8D] 1 is a graph illustrating a set of B-spline basis functions of orders 0 to 3 for a uniform distribution of knot points according to an example. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present disclosure and aspects thereof may be embodied in various forms, including hardware, devices, or circuits controlled by computer-implemented methods, computer program products, computer systems and networks, user interfaces, application programming interfaces, and hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to give a general idea of ​​various aspects of the present disclosure and is not intended to limit the scope of the present disclosure in any way.

[0013] In the following description, numerous details are set forth, such as the configuration, timing, operation, etc. of optical devices, in order to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to those skilled in the art that these specific details are merely exemplary and are not intended to limit the scope of the present application.

[0014] The various methods used to compress images typically fall into one of two categories: lossy and lossless. Lossy compression reduces image file size by permanently removing some information, which is irreversibly lost. A prime example of lossy compression is the JPEG format, widely used on the web and in digital photography. Lossless compression reduces image file size without removing important data, resulting in a compressed image that can be restored to its original state without degradation or distortion. However, lossless compression typically does not reduce file size as much as lossy compression. As a result, lossless compression is typically used in situations where image quality is more important than disk space or network performance, such as product images or artwork displays.

[0015] As used herein, the term "dynamic range" (DR) may refer to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from darkest black (dark) to brightest white (highlight). In this sense, DR relates to "scene-referred" intensities. DR may also refer to the ability of a display device to fully or approximately render a particular width of intensity range. In this sense, DR relates to "display-referred" intensities. At any point in this specification, the term may be used in either sense (e.g., interchangeably) unless one particular sense is expressly stated to be particularly important.

[0016] As used herein, the term "high dynamic range" (HDR) refers to a DR width spanning 14-15 orders of magnitude or more in HVS. In practice, the DR that humans can simultaneously perceive across a wide intensity range may be somewhat truncated for HDR. As used herein, the terms "enhanced dynamic range" (EDR) or "visual dynamic range" (VDR) may individually or interchangeably refer to the DR perceivable in a scene or image by the human visual system, including eye movements, that allows for some degree of light adaptation change across the scene or image. As used herein, EDR may refer to a DR spanning 5-6 orders of magnitude. While perhaps somewhat narrower than true scene-referred HDR, EDR represents a wide DR width and may also be referred to as HDR.

[0017] In practice, an image has one or more color components of a color space (e.g., luma Y and chroma Cb and Cr), each color component represented with n bits of precision per pixel (e.g., n=8). When using non-linear luminance coding (e.g., gamma coding), an image with n≦8 (e.g., a color 24-bit JPEG image) can be considered a standard dynamic range (SDR) image, and an image with n>8 can be considered an EDR image.

[0018] In the mathematical field of numerical analysis, a B-spline, or basis spline, is a spline function that has minimum support for a given degree, smoothness, and domain division. Any spline function of a given degree can be expressed as a linear combination of B-splines of that degree. Cardinal B-splines have knots that are equidistant from one another. B-splines can be used, for example, for curve fitting and numerical differentiation of experimental data. In computer-aided design and computer graphics, spline functions are constructed as linear combinations of B-splines with a set of control points.

[0019] One-dimensional B-splines can be extended to multiple dimensions by using tensor product B-spline (TPB) structures. Multivariate basis functions are constructed by multiplying B-splines corresponding to different dimensions. For example, first, a B-spline subspace is defined for each dimension. Then, a tensor product function in the multidimensional space is constructed as a weighted sum of one-dimensional B-spline structures, with the weights representing corresponding tensor elements. A brief description of the main TPB concepts relevant to various embodiments disclosed herein is provided at the end of this specification.

[0020] Example of a video / image distribution pipeline 1 is a block diagram illustrating an example process for a video / image delivery pipeline (100) showing various stages from video / image capture to video / image content display according to one embodiment. A sequence of video / image frames (102) may be captured or generated using an image generation block (105). The frames (102) may be digitally captured (e.g., by a digital camera) or computer-generated (e.g., using computer animation) to provide video and / or image data (107). Alternatively, the frames (102) may be captured on film by a film camera. The film may then be converted to a digital format to provide the video / image data (107).

[0021] In the production phase (110), the data (107) may be edited to provide a video / image production stream (112). Data from the video / image production stream (112) may be provided to a processor (or one or more processors, such as a central processing unit (CPU)) in a post-production block (115) for post-production editing. Post-production editing in block (115) may include, for example, adjusting or correcting the color or brightness of specific areas of an image to enhance image quality or achieve a particular look for the image according to the filmmaker's creative intent. This part of post-production editing is sometimes referred to as "color timing" or "color grading." Other edits (e.g., scene selection and sequencing, image cropping, adding computer-generated visual special effects, removing artifacts, etc.) may also be performed in block (115) to produce a "final" version (117) of the work for distribution. During post-production editing (115), the video and / or images may be viewed on a reference display (125).

[0022] Following post-production (115), the final version (117) of the data may be delivered to an encoding block (120) for further downstream distribution to decoding and playback devices such as television sets, set-top boxes, flat or curved displays, personal computers, mobile devices, head-mounted displays, etc. In some embodiments, the encoding block (120) may include audio and video encoders, such as those specified by ATSC, DVB, DVD, Blu-Ray, and other distribution formats, to generate the encoded bitstream (122). In some examples, the electronic encoder used in the encoding block (120) comprises or is implemented using an electronic processor coupled to a memory containing program code. The memory and program code are configured to cause the electronic processor to perform various encoding operations, such as non-limiting examples of the various encoding operations described below with reference to FIG. 2.

[0023] After transmission via a distribution or delivery channel, the encoded bitstream (122) is decoded in a decoding block (130) to generate a corresponding decoded signal (132) representing a copy or approximation of the signal (117). The decoding block (130) is communicatively connected to a target display (140), which may have characteristics that are slightly or completely different from the reference display (125). In such cases, a display management (DM) block (135) may be used to map the decoded signal (132) to the characteristics of the target display (140) by generating a display map signal (137). Depending on the embodiment, the decoding unit (130) and the display management block (135) may include separate processors or may be based on a single integrated processing unit. In some examples, the electronic decoder used in the decoding block (130) comprises or is implemented using an electronic processor coupled to a memory containing program code. The memory and program code are configured, using the electronic processor, to cause the electronic decoder to perform various decoding operations, such as non-limiting examples of the various decoding operations described below with reference to FIG.

[0024] The codecs used in the encoding block (120) and / or decoding block (130) allow for the processing and compression / decompression of video / image data. Compression is used in the encoding block (120) to make the corresponding file or stream smaller. The decoding process performed by the decoding block (130) typically involves decompressing the received video / image data file or stream into a form usable for playback and / or further editing. Examples of encoding / decoding operations usable in the encoding block (120) and decoding unit (130) according to various embodiments are described in more detail below.

[0025] Encoder and decoder architecture In some examples, every color pixel in a digital image is created by a combination of three primary colors, such as red, green, and blue (RGB). Each primary color is typically called a "color channel" or "color component" and has a range of intensity values ​​specified by its bit depth. The bit depth of each primary color is called "bits per channel" and typically ranges from 8 to 16 bits. "Bits per pixel" (bpp) refers to the sum of the "bits per color channels," i.e., the total number of bits used to encode the color information of a pixel. For example, an uncompressed RGB image with a bit depth of 8 bits per color has 24 bpp, or 24 bits per pixel (which includes 8 bits for red, 8 bits for green, and 8 bits for blue). Standard color conversions can be used to convert RGB images to YCbCr color format and vice versa.

[0026] Some embodiments disclosed herein can be used to compress multimedia, such as color images, using a TPB representation. Unlike some conventional compression methods that operate on digitized samples such as pixels, various embodiments are configured to explore the inherent structure of the image and reduce redundancy by modeling the image using a continuous function and performing optimization on the function coefficients. In some examples, to represent the image, an encoder operates to adapt an implicit TPB model using input coordinates to construct a continuous function that maps digitized sampled pixel locations to corresponding digitized sampled pixel values. The TPB coefficients are compressed via quantization and entropy coding and transmitted to a decoder. The decoder first applies decompression to reconstruct the TPB coefficients and then operates to query the pixel locations with the corresponding TPB function to generate pixel values. In at least some examples, a spatial domain coordinate reshaping method is used to improve performance. At least some embodiments can beneficially outperform some conventional image compression techniques, such as JPEG, at relatively low bit rates, e.g., below about 0.5 bpp. Furthermore, unlike some typical neural network-based solutions, the TPB-based solutions disclosed herein do not typically rely on iterative calculations such as backpropagation, beneficially enabling implementations with low computational power and small memory allocations.

[0027] Figure 2 is a block diagram illustrating an encoder (200) according to one embodiment. The hardware of the encoder (200) includes an electronic processor (270) and a memory (280) operatively coupled to each other. The electronic processor (270) and the memory (280) enable the various modules of the encoder (200) to perform various encoding operations, as described below. An example of an input to the encoder (200) includes an RGB image (202). The encoder (200) processes the image (202) to generate a metadata bitstream (298). In some examples, the encoder (200) is used in the encoding block (120) of the pipeline (100). The image (202) is part of the stream (117). The metadata bitstream (298) forms part of the encoded bitstream (122).

[0028] In the illustrated example, the encoder (200) primarily operates in the YCbCr color domain. To this end, the encoder (200) includes an RGB-to-YCbCr converter (210) configured to convert an input RGB image (202) into a corresponding YCbCr image (212). In some examples, the RGB-to-YCbCr conversion implemented in the RGB-to-YCbCr converter (210) is a simple, standard operation known to those skilled in the art. In some other examples, the encoder (200) operates directly in the RGB color domain. In such examples, the RGB-to-YCbCr converter (210) is omitted or bypassed.

[0029] In some examples, the YCbCr image (212) generated by the RGB-to-YCbCr converter (210) has a YCbCr 4:4:4 format. Because the encoder (200) is configured to use a respective coordinate-based TPB model for each color channel, in such examples, no chroma subsampling is performed. By omitting the chroma subsampling, relatively small adjustments to the coordinate model itself (fewer parameters than the original size) are required, while saving computational power. In other examples, other YCbCr formats, such as a YCbCr 4:2:2 format or a 4:2:0 format, are similarly used. In such other examples, a chroma subsampling operation may be performed.

[0030] The encoder (200) implements a patch-based solution in which each patch is modeled by a different implicit TPB function. In a representative example, the encoder (200) splits the image (212) into patches, e.g., patches of size W p ×H p It works by dividing the image into non-overlapping rectangular patches where W p is the patch width and H p is the patch height. Each patch of the image (212) is processed in the encoder (200) using the same sequence of operations, for example, as described in more detail below. Within that sequence, the operations of the spatial reshaping module (220) are applied to the entire image (212), while the modeling and optimization operations of the downstream modules (240, 250) are applied to each patch individually.

[0031] The spatial reshaping module (220) operates on the YCbCr image (212) to perform spatial domain coordinate reshaping and generate a corresponding spatially reshaped signal (222). Typically, the image (212) exhibits different characteristics in different local regions. For example, different regions of the image (212) may exhibit different distributions of texture frequencies. Given that the TPB model has a limited capacity to model a region with a certain accuracy, having a more uniform distribution of texture frequencies along each dimension can more efficiently utilize the capacity of the TPB model. In other words, algorithmic methods for redistributing texture frequencies can improve the accuracy of the model within the same parameter budget. Based on this observation, the encoder (200) is configured to perform spatial domain coordinate reshaping (220) that maps the regular (constant) spacing of the sampled grid coordinates to a non-uniform spacing so that local image complexity (e.g., as assessed using the local standard deviation) is more uniform, facilitating subsequent TPB modeling with greater accuracy. As already indicated, the spatial reshaping function applied in module (220) is constructed over the entire image (212) along each dimension separately. The spatial reshaping function applied in module (220) is then modeled by its own TPB function in modeling block (240) to generate a spatially reshaped signal model (242). Sending the TPB model (242) to the decoder instead of the underlying reshaped signal (222) allows for a simultaneous reduction in the corresponding bitrate overhead in the metadata bitstream (298).

[0032] The patch extraction module (230) operates to extract individual patch signals (232) using the spatially reshaped signal (222). The TPB optimization module (250) then operates to calculate optimized TPB models for different patches of the image (212). More specifically, each optimized TPB model for each patch is calculated in the module (250) by finding parameter values ​​for that model such that the difference between each image patch signal (232) and the reconstructed image patch signal (244), calculated using the TPB model (242), is approximately minimized. In some examples, rate-distortion (RD) optimization can be deployed to achieve near-maximum quality of the reconstructed image at the decoder for a target bitrate.

[0033] The metadata generation module (260) operates to generate a metadata bitstream (298) based on (i) a parameter set (256) having coefficients of the TPB model (242) and (ii) a plurality of parameter sets (258). Each parameter set (258) includes coefficients of the TPB function for each patch generated by the TPB optimization module (250). The parameter sets (258) are compressed by the metadata generation module (260) via nonlinear quantization and arithmetic coding. The resulting compressed parameter sets, along with side information for the nonlinear quantization and arithmetic coding, are transmitted via the metadata bitstream (298). The parameter sets (256) are also transmitted via the metadata bitstream (298).

[0034] FIG. 3 is a block diagram illustrating a decoder (300) according to one embodiment. The decoder (300) hardware includes an electronic processor (370) and a memory (380) operatively coupled to each other. The electronic processor (370) and the memory (380) enable the various modules of the decoder (300) to perform various decoding operations, as described below. An example of an input to the decoder (300) includes a metadata bitstream (298) received from the encoder (200). The decoder (300) processes the metadata bitstream (298) to generate a reconstructed RGB image (302). In a typical example, the reconstructed RGB image (302) is an approximation of the RGB image (202). In some examples, the decoder (300) is used in the decoding block (130) of the pipeline (100). In such examples, the image (302) is part of the decoded signal (132).

[0035] The metadata extraction module (360) operates to parse the received metadata bitstream (298) into (i) a parameter set (256) having coefficients of the TPB model (242) and (ii) a plurality of compressed parameter sets (358). Each compressed parameter set (358) is a compressed version of a respective parameter set (258) generated by the TPB optimization module (250) in the encoder (200). The modeling block (340) generates a spatially reshaped signal (338) using the TPB model (242) configured with the parameter sets (256). The decompression module (350) operates to decompress the compressed parameter sets (358) to approximately reconstruct the parameter sets (258) via arithmetic decoding and inverse quantization configured using applicable side information received via the metadata bitstream (298). Typically, the reconstruction of the parameter set (258) in the decompression module (350) is not exact due to the lossy compression used in the encoder (200). For this reason, the output of the decompression module (350) is denoted as parameter set (258') to indicate the typical deviation from the parameter set (258). The decoder (300) uses the parameter set (258') to construct a corresponding patch-wise TPB model (348).

[0036] The patch generation module (330) uses the spatial reshaping signal (338) to provide the reshaped coordinates as input to a patch-wise TPB model (348) to generate a reconstructed image patch signal (328). This procedure is repeated by the patch generation module (330) for each color channel of each patch. After the patch generation module (330) has completed calculations for the Y, Cb, and Cr color channels of all patches, the YCbCr image construction module (320) assembles the reconstructed patches into a reconstructed YCbCr image (318). In a typical example, the reconstructed YCbCr image (318) is an approximation of the YCbCr image (212). Finally, the YCbCr-to-RGB converter (310) operates to convert the YCbCr image (318) into an RGB image (302).

[0037] Various examples of different operations performed in the encoder (200) and decoder (300) are described in more detail in the four subsections that follow this subsection.

[0038] Frame-based spatial domain coordinate reshaping (spatial reshaping) A typical image processed by the encoder (200) and decoder (300) exhibits significant variation in its different local regions. As a result, "content complexity" can vary significantly across various regions of the image. This variability is addressed by the patch-based solution described above. The "model capacity" in each region is typically limited by the TPB model rank. Therefore, in addition to adaptively selecting the TPB model rank for each region, further performance improvements are achieved by "tuning" (reshaping) the content complexity in the spatial domain via nonlinear coordinate mapping. In other words, the content complexity can be reshaped so that each local region (e.g., each patch) can have more uniformity in terms of content complexity.

[0039] In some examples, "content complexity" is defined based on the standard deviation from the mean. Thus, if the standard deviation of a first image portion is greater than the standard deviation of a second image portion, the first image portion is considered to have higher "content complexity" than the second image portion. The standard deviation is considered a linear measure of "content complexity." In some examples, the goal of spatial reshaping is to create a coordinate mapping function that can shift a non-uniform distribution of local content complexity to be more uniform.

[0040] The width and height of the input image (I) are denoted as W and H, respectively. A pixel (x, y) in the image can be represented as I(x, y). The spatial coordinates (x, y) are normalized to the range [0, 1].

[0041] The spatial domain coordinate reshaping functions along the x-axis (horizontal) and y-axis (vertical) of the spatial domain coordinates are denoted by γ h and γ v The new mapped spatial coordinates corresponding to (x, y) are

number

number

[0042] W centered at (x, y) L ×H L Let the content complexity in the local region (denoted as N(x,y)) be denoted as σ(x,y). (W L , H L ) is typically (51, 51). For the 2D standard deviation as a measure of content complexity, σ(x,y) can be calculated as follows:

number

number

[0043] A desirable reshaping function is one that results in a more uniform local standard deviation after reshaping. In some examples, a 1D function is used for each dimension to simplify the design and signaling of the reshaping function in the bitstream (298). Because the local standard deviation is measured in the corresponding 2D local region, the average standard deviation can be calculated for each column and each row. Taking the x-axis as an example, first, for all pixels with horizontal coordinate x, the average standard deviation can be calculated as follows:

number

number

number

[0044] Again, taking the x-axis as an example, the goal of spatial reshaping is to find a mapping function γ x The goal is to find a function () such that the local standard deviation (STD) in the new grid {x'} is more uniform. In some examples, the goal is to find a function with the following properties:

number

number

number

number

[0045] Figures 4A and 4B graphically illustrate a 1D spatial reshaping function calculated in an example encoder (200). The image (202) corresponding to this example is a color portrait of a woman wearing a red hat. Figure 4A illustrates a graph of the linear coordinate mapping function (402), i.e., without spatial reshaping. Figure 4B illustrates a graph of the spatial domain coordinate reshaping function (404) calculated in the encoder (200) for the example image (202) based on the local STD measurement described above. At the same bit rate, use of the reshaping function (404) improves PSNR by approximately 0.3 dB (measured in RGB color space for three color channels).

[0046] The spatial reshaping module (220) uses a spatial reshaping function, such as the example spatial reshaping function (404), to convert the regular (uniformly sampled) grid locations to non-uniformly spaced locations, e.g., as described above. This spatial reshaping function is used by downstream modules of the encoder (200) and further used in the decoder (300), as described above. To reduce the overhead associated with transmitting the spatial reshaping function from the encoder (200) to the decoder (300), the encoder (200) is configured to compress the spatial reshaping function using the corresponding TPB model generated by its modeling block (240). The following description details the operations used to compress the spatial reshaping function based on the 1D TPB model.

[0047] For example, in the case of x-axis modeling, the spatial reshaping function generates W input / output data pairs.

number

number

[0048] Given a set of selected knot points T (where T is the number of knots), a B-spline order l (where l is the degree), uniform spacing, and a corresponding D sr,h (=T+l-1) basis function

number

number

number

number

number

[0049] The correct vector is shown.

number

number

number

number

number

number

number

[0050]

number

number

number

number

[0051] In some cases, the set

number

number

number

number

number

[0052] Optimizing implicit TPB functions per patch The operations described in the previous section apply spatial domain coordinate reshaping to the entire image to readjust local image complexity. In contrast, this section describes patch-wise modeling that obtains a separate implicit TPB (iTPB) model for each patch of the image. More specifically, for each color channel of each image patch, the encoder (200) operates to generate a respective individual TPB model characterized by a corresponding set of TPB coefficients. The encoder (200) also operates to perform optimization of these individual TPB models to adjust the TPB coefficients to meet an optimization criterion.

[0053] An implicit function accepts coordinates (rather than pixel values) as input. One example of an implicit function is a neural field implemented using a neural network. In some examples, the neural field employs a multi-layer perceptron (MLP) to model (or explore) continuous functions, the training of which can take a relatively long time if it relies on a multi-iteration solver (e.g., backpropagation). Various embodiments disclosed herein use a TPB to implement the implicit function. The input to the TPB is coordinates. Using ground truth data, a non-iterative least-squares solution (a deterministic approach) is used to determine the TPB configuration. As a result, finding a set of coefficients suitable for the TPB model can be computationally lighter and faster than that of a functionally equivalent neural field.

[0054] Consider the kth patch of an image, taking the luma channel as an example. For this patch, Φ k = {(x,y)} the original input pixel coordinates and the corresponding spatially reshaped coordinates calculated as above

number

number

number

number

number

number

number

[0055] In some examples, the TPB basis functions for predicting the luma channel are constructed by multiplying two sets of basis functions as follows:

number

number

number

[0056] In some cases, the 2D index

number

number

number

number

number

number

[0057] The ground truth vector using the original coordinates {(x,y)} for the reference base layer luma is expressed as follows:

number

number

number

number

number

[0058] For the chroma channels, follow the same steps,

number

number

[0059] Implicit TPB coefficient compression FIG. 5 is a block diagram illustrating a TPB coefficient compression operation (500) performed in a metadata generation module (260) according to one embodiment. Input to the compression operation (500) includes, for example, a parameter set (258) having optimized iTPB coefficients calculated as described above. In the illustrated example, the compression operation includes blocks of operations (510, 520). The first block (510) operates to apply nonlinear quantization to the iTPB coefficients. The second block (520) operates to apply arithmetic coding to compress the resulting quantized TPB coefficients (512), thereby generating a corresponding stream of compressed iTPB coefficients (522). Note that the compression operation of the first block (510) is lossy, and the configuration of these operations typically controls the amount of information loss. The compression operation of the second block (520) is lossless, and the configuration of these operations typically controls the bitrate. To achieve an optimal rate-distortion (RD) combination, the settings of both blocks (510, 520) can be adjusted, for example, as described in the RD optimization subsection below. The data stream (522) produced by the compression operation (500) is sent to the decoder (300) along with the metadata bitstream (298).

[0060] The implicit TPB function coefficients at the kth patch for color channel c are

number

[0061] FIG. 6 shows a bar graph (602) illustrating the distribution of uniformly quantized 6-bit TPB coefficient values ​​according to an example. This example corresponds to the entire first Kodak dataset image. As evident from the bar graph (602), the distribution is non-uniform and biased toward positive values. More specifically, most of the coefficients are positive and clustered at low magnitudes. It is also observed from the bar graph (602) that the distribution is substantially contained within the [0,1] range, with only a small fraction of values ​​outside the [0,1] range. Based on these two observations, the block (510) is configured to apply a non-linear quantization (NLQ) operation to give higher priority to values ​​located within [0,1] compared to other values. Note that the maximum magnitude of the TPB coefficients is unlimited because the coefficient values ​​are directly output from the least-squares algorithm. Thus, the NLQ used in the block (510) is configured to accommodate the relatively large dynamic range available for the TPB coefficients. In one example, the NLQ used in the operational block (510) is implemented using log-domain quantization.

[0062] In some examples, block (510) first:

number

number

number

number

number

number

number

number

number

[0063] The decoder (300) processes the quantized coefficients

number

number

number

number

number

number

[0064] Patch size (W p , H p ) for the kth patch with the number of TPB basis functions

number

number

number

[0065] After quantizing the TPB coefficients in a block (510), the compression operation proceeds to arithmetically coding the block (520),

number

[0066] The number of bits after applying arithmetic coding to the quantized TPB coefficients in each patch is

number

number

number

[0067] 7A and 7B are block diagrams illustrating an example format of a metadata stream (298) according to one embodiment. In the illustrated example, the metadata stream (298) is configured to transmit a compressed color image (202). Accordingly, the metadata stream (298) includes three corresponding portions (702, 704, 706). The first portion (702) includes data representing the Y channel. The second portion (704) includes data representing the Cb channel. The third portion (706) includes data representing the Cr channel. For purposes of illustration, and without implying any limitation, only the format of the first portion (702), representing the Y channel, is explicitly shown in FIGS. 7A and 7B. In some examples, the second portion (704) and third portion (706), representing the Cb and Cr channels, respectively, have the same format as the first portion (702). The first portion (702) is partitioned to include a first portion (712) and a second portion (714). The first part (712) contains data representing the TPB model (242), i.e., the TPB model of the spatial reshaping function. The second part (714) contains data representing the patch-wise TPB model parameter set (258). Figure 7A shows the first part (712) in more detail. Figure 7B similarly shows the second part (714) in more detail.

[0068] Referring to FIG. 7A , the first part (712) of the luma portion (702) includes metadata (722) representing the spatial reshaping signal (222) in the y-direction and further includes metadata (724) representing the spatial reshaping signal (222) in the x-direction. In some examples, the rank and number of knots of the TPB model are fixed. In such examples, transmission of the TPB model of the spatial reshaping function may be achieved by transmitting a fixed number (e.g., six) of TPB coefficients per direction. As such, the y-direction metadata (722) includes a first set (732) of such TPB coefficients and is explicitly shown in FIG. 7A . The x-direction metadata (724) includes a second set (not explicitly shown in FIG. 7A ) of such TPB coefficients. In some examples, the metadata (722, 724) includes 12 floating-point coefficient values.

[0069] Referring to FIG. 7B, the second part (714) of the luma portion (702) includes a patch dimension field (742) and multiple patch-level metadata fields (744-1, ..., 744-N). The patch dimension field (742) includes patch width and patch height values ​​W and H. Each of the patch-level metadata fields (744-n) (where n = 1, ..., N) includes a respective set of subfields (750-764). For illustrative purposes, only the set of subfields (750-764) corresponding to the patch-level metadata field (744-1) is explicitly shown in FIG. 7B. Each of the other patch-level metadata fields (744-n) has a similar set of subfields (750-764) (not explicitly shown in FIG. 7B).

[0070] The first subfield (750) specifies the number of TPB knots for the patch. The second subfield (752) specifies the bitstream encoding mode. The third subfield (754) specifies whether the coefficients are non-zero (NZ) or zero (Z). In the uncompressed mode 0 format, the third subfield (754) uses one bit to indicate the type of zero / non-zero value for each coefficient. In the compressed mode 1 format, the third subfield (754) uses run-length encoding (NZ / Z, run). For each run-length symbol, one bit is used for NZ / Z and eight bits are used to indicate RUN. The encoder (200) selects the smaller of the two modes. The fourth subfield (756) uses a one-bit syntax to indicate whether the coefficients are positive or negative. In the uncompressed format of mode 0, the fourth subfield (756) uses one bit to indicate whether the sign of each coefficient is positive (P) or negative (N). If the coefficient is zero, no signal is needed from the NZ / Z flag in the third subfield (754). In the compressed format of mode 1, the fourth subfield (756) uses run-length encoding (P / N, run). For each run-length symbol, one bit is used for P / N and eight bits are used to indicate RUN. The encoder selects the smaller of the two modes. The fifth subfield (758) signals the number of quantization bits. The sixth subfield (760) signals the number of quantization bits.

number

[0071] Rate-Distortion (RD) Optimization In some examples, several algorithm parameters are adjusted to optimize PSNR or bitrate. These parameters include: Patch size (Wp , H p ) Number of basis functions

number

[0072] Target bitrate R T Using , this RD optimization problem can be formulated as an optimization problem that minimizes distortion (eg, maximizes PSNR), for example, as shown below:

number

number

number

number

number

[0073] In some instances, a full search algorithm is deployed to find the optimal solution, for example, based on the following pseudocode:

number

number

[0074] Note that the above full search takes a relatively long time and may be impractical for some applications. From experiments and simulations, we have found that in some examples, fixed values ​​of Q=6 and W p =256, H p = 256 can be used to search for the optimal solution.

number

number

[0075] In some examples, RD optimization can be further sped up using a greedy algorithm. For every patch, the algorithm starts by providing the same number of knots. Then, at each iteration, the algorithm increases the delta number of knots in each patch.

number

number

[0076] According to the example embodiments disclosed above, e.g., in the Overview section and / or with reference to any one of Figures 1-8 or any combination of some or all of Figures 1-8, a method for compressing images is provided. A method for compressing an image includes generating, using an electronic encoder, a first tensor product B-spline (TPB) model representing the image, the first TPB model approximating a spatial reshaping function configured to map uniformly sampled locations in the image to non-uniformly spaced locations; generating, using the electronic encoder, a plurality of second TPB models, each of the plurality of second TPB models representing a respective patch of the image and configured to estimate image signals in each patch as a function of reshaped coordinates, the reshaped coordinates being determined using the spatial reshaping function; and generating, using the electronic encoder, a metadata stream including first metadata representing a set of coefficients of the first TPB model and second metadata representing a set of coefficients of the plurality of second TPB models.

[0077] In some embodiments of the above method, generating the second metadata includes compressing the set of coefficients of the plurality of second TPB models using an electronic encoder.

[0078] In some embodiments of any one of the above methods, the compressing includes generating a set of quantized coefficients by quantizing each coefficient to a fewer number of bits, and applying arithmetic coding to the set of quantized coefficients.

[0079] In some embodiments of any one of the above methods, the metadata stream includes side information characterizing at least one of the quantizing and the arithmetic coding.

[0080] In some embodiments of any one of the above methods, the method further includes dividing the image into a plurality of non-overlapping patches, using an electronic encoder, each patch selected from the plurality of non-overlapping patches.

[0081] In some embodiments of any one of the above methods, the image is a YCbCr image, and the method further comprises generating the YCbCr image by applying an RGB-to-YCbCr transform to the RGB image.

[0082] In some embodiments of any one of the above methods, the plurality of second TPB models includes a plurality of luma channel TPB models and a plurality of chroma channel TPB models.

[0083] In some embodiments of any one of the above methods, the spatial reshaping function is configured to shift a non-uniform distribution of local content complexity within the image to make it more uniform.

[0084] In some embodiments of any one of the above methods, the two-dimensional standard deviation is a measure of local content complexity.

[0085] In some embodiments of any one of the above methods, generating the plurality of second TPB models includes finding, for a second TPB model from the plurality of second TPB models, a respective set of coefficients that approximately minimizes a difference between the image signal and an estimate of the image signal in each patch, wherein the estimate of the image signal is calculated using the second TPB model.

[0086] In some embodiments of any one of the above methods, the finding is based on least-squares minimization.

[0087] In some embodiments of any one of the above methods, the method further comprises transmitting the metadata stream to the electronic decoder via a communication channel.

[0088] In some embodiments of any one of the above methods, the reshaped coordinates are normalized to be in the range [0,1].

[0089] In some embodiments of any one of the above methods, the compressing includes applying entropy coding to the set of quantized coefficients.

[0090] In some embodiments of any one of the above methods, the entropy coding comprises one of arithmetic coding and Huffman coding.

[0091] In some embodiments of any one of the above methods, the quantizing includes applying non-linear quantization to coefficients of the plurality of second TPB models.

[0092] In some embodiments of any one of the above methods, the image compression method further includes performing rate-distortion optimization to select one or more parameters from the group consisting of a patch size, a number of basis functions, and a number of bits for the quantized TPB coefficients.

[0093] In some embodiments of any one of the above methods, the rate-distortion optimization is based on a greedy algorithm.

[0094] In some embodiments of any one of the above methods, the spatial reshaping function is a normalized cumulative function of local standard deviations calculated via a process of standard deviation equalization (e.g., as represented by equations (10) and (11)).

[0095] According to another example of the embodiments disclosed above, e.g., in the Overview section and / or with reference to any one of Figures 1-8 or any combination of some or all of Figures 1-8, an apparatus for image compression is provided. An apparatus for image compression comprises at least one processor and at least one memory containing program code, wherein the at least one memory and the program code are configured, using the at least one processor, to cause the apparatus to at least: generate a first tensor product B-spline (TPB) model representing an image, the first TPB model approximating a spatial reshaping function configured to map uniformly sampled locations in the image to non-uniformly spaced locations; generate a plurality of second TPB models, each of the plurality of second TPB models representing a respective patch of the image and configured to estimate image signals in each patch as a function of reshaped coordinates, the reshaped coordinates being determined using the spatial reshaping function; and generate a metadata stream containing first metadata representing a set of coefficients of the first TPB model and second metadata representing a set of coefficients of the plurality of second TPB models.

[0096] According to another example of the embodiments disclosed above, e.g., in the Overview section and / or with reference to any one of Figures 1-8 or any combination of some or all of Figures 1-8, there is provided a method for decompressing an image. A method for image decompression includes receiving, with an electronic decoder, a metadata stream including first metadata representing a set of coefficients of a first tensor product B-spline (TPB) model and second metadata representing sets of coefficients of a plurality of second TPB models, the first TPB model approximating a spatial reshaping function configured to map uniformly sampled locations in a source image to non-uniformly spaced locations, and each of the second TPB models representing a respective patch of the source image and configured to estimate image signals in each patch as a function of reshaped coordinates, the reshaped coordinates being determined based on the spatial reshaping function; generating, with the electronic decoder, a plurality of reconstructed image patches based on the first TPB model and further based on a plurality of second TPB models, the first TPB model being reconstructed using the first metadata and the plurality of second TPB models being reconstructed using the second metadata; and constructing, with the electronic decoder, an estimated image by assembling the plurality of reconstructed image patches in an image frame.

[0097] Some embodiments of the above method further include decompressing, with an electronic decoder, the second metadata to reconstruct the set of coefficients of the plurality of second TPB models.

[0098] In some embodiments of any one of the above methods, the decompressing is based on side information included in the metadata stream, the side information characterizing quantization and arithmetic coding of at least one of the sets of coefficients of the plurality of second TPB models.

[0099] In some embodiments of any one of the above methods, the plurality of second TPB models includes a plurality of luma channel TPB models and a plurality of chroma channel TPB models.

[0100] In some embodiments of any one of the above methods, the spatial reshape function is configured to shift a non-uniform distribution of local content complexity in the source image to be more uniform.

[0101] In some embodiments of any one of the above methods, the two-dimensional standard deviation is a measure of local content complexity.

[0102] According to another example of the embodiments disclosed above, e.g., in the Summary section, and / or with reference to any one of Figures 1-8 or any combination of some or all of Figures 1-8, there is provided an apparatus for image decompression, the apparatus comprising at least one processor and at least one memory including program code, the at least one memory and the program code configured to, using the at least one processor, cause the apparatus to: receiving a metadata stream including first metadata representing a set of coefficients of a TPB (Typical B-spline) model and second metadata representing sets of coefficients of a plurality of second TPB models, where the first TPB model approximates a spatial reshaping function configured to map uniformly sampled locations in a source image to non-uniformly spaced locations, and where each of the second TPB models represents a respective patch of the source image and is configured to estimate an image signal in each patch as a function of reshaped coordinates, the reshaped coordinates being determined using the spatial reshaping function; generating a plurality of reconstructed image patches based on the first TPB model and further based on the plurality of second TPB models, where the first TPB model is reconstructed using the first metadata and the plurality of second TPB models are reconstructed using the second metadata; and constructing an estimated image by assembling the plurality of reconstructed image patches in an image frame.

[0103] With respect to processes, systems, methods, heuristics, etc. described herein, although steps of such processes, etc., are described as occurring according to a particular order, it should be understood that such processes may be practiced by performing the steps described herein in an order other than the order described herein. Furthermore, it should be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the process descriptions herein are provided for the purpose of describing particular embodiments and should not be construed as limiting the scope of the claims in any way.

[0104] Therefore, it should be understood that the above description is illustrative and not restrictive. Many embodiments and applications other than the examples provided will become apparent upon reading the above description. The scope of the disclosure should be determined not with reference to the above description, but with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In short, it should be understood that this application is capable of modification and variation.

[0105] All terms used in the claims are intended to be given their broadest reasonable construction and their ordinary meaning as understood by those skilled in the art described herein, unless expressly indicated to the contrary. In particular, the use of singular articles such as "a," "the," "said," etc., should be read as describing one or more of the indicated elements, unless an express to the contrary limitation is recited in the claims.

[0106] The Abstract of the present disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the above Detailed Description, various features may be seen grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments include more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

[0107] While this disclosure includes reference to exemplary embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments that are apparent to those skilled in the art to which this disclosure pertains, as well as other embodiments within the scope of this disclosure, are deemed to be within the principles and scope of the disclosure, as expressed, for example, in the following claims.

[0108] Some embodiments may be implemented as circuit-based processes, including potentially implementation on a single integrated circuit.

[0109] Some embodiments may be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments may also be embodied in the form of program code recorded on tangible media, such as magnetic recording media, optical recording media, solid-state memory, floppy diskettes, CD-ROMs, hard drives, or any other non-transitory machine-readable storage media. When the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the patented invention. Some embodiments may also be embodied in the form of program code stored on a non-transitory machine-readable storage medium, for example, including being loaded into and / or executed by a machine. When the program code is loaded into and executed by a machine, such as a computer or processor, the machine becomes an apparatus for practicing the patented invention. When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0110] Unless expressly stated otherwise, each numerical value and range should be construed as approximate, as if the word "about" or "approximately" were preceding the value or range.

[0111] The use of figure numbers and / or figure reference signs in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate claim interpretation, and such use should not necessarily be construed as limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0112] In the method claims below, elements, if any, are described in a particular sequence with corresponding labeling, but the elements are not necessarily intended to be limited to being implemented in that particular sequence, unless the claim recitation otherwise suggests a particular sequence for implementing some or all of the elements.

[0113] As used herein, "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of the present disclosure. The appearances of the phrase "in one embodiment" in various places in this specification do not necessarily all refer to the same embodiment, nor do they necessarily refer to separate or alternative embodiments that are mutually exclusive of other embodiments. The same applies to the term "implementation."

[0114] Unless otherwise specified herein, the use of ordinal adjectives "first," "second," "third," etc. to refer to multiple similar objects merely indicates that different instances of such similar objects are being referenced, and is not intended to imply that the similar objects so referenced must be in a corresponding order or sequence in time, space, precedence, or otherwise.

[0115] Unless otherwise specified herein, the conjunction "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting," in addition to its plain meaning, and these interpretations may depend on the corresponding specific context. For example, the phrase "when determined" or "when (a referenced condition) is detected" may be interpreted as "upon determining" or "in response to determining" or "upon detecting (a referenced condition or event)" or "in response to detecting (a referenced condition or event)."

[0116] As used herein with respect to elements and standards, the term compatible means that an element communicates with other elements in a manner specified in whole or in part by the standard and is recognized by other elements as being sufficiently capable of communicating with them in the manner specified by the standard. A compatible element need not operate internally in the manner specified by the standard.

[0117] The functions of the various elements shown in the figures, including any functional blocks labeled "processor" and / or "controller," may be provided through the use of dedicated hardware as well as hardware capable of executing software in conjunction with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. Furthermore, explicit use of the terms "processor" or "controller" should not be construed as referring exclusively to hardware capable of executing software, but may implicitly include digital signal processor (DSP) hardware, network processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. The functions of those switches may be performed by the operation of program logic, by dedicated logic, by the interaction of program control and dedicated logic, or even manually, with the particular technique being selectable by the implementer, as more particularly understood from the context.

[0118] As used in this application, the term “circuitry” may refer to one or more or all of the following: (a) a hardware-only circuit implementation (such as an implementation in only analog and / or digital circuitry); (b) a combination of hardware circuitry and software, such as (where applicable): (i) a combination of analog and / or digital hardware circuitry and software / firmware; and (ii) a hardware processor (including a digital signal processor) with software, software, and any portion of memory that cooperate to cause a device such as a cell phone or server to perform various functions; and (c) a hardware circuit and / or processor, such as a microprocessor or portion of a microprocessor, that requires software (e.g., firmware) for operation but may not be present when not required for operation. This definition of circuitry applies to all uses of this term in this application, including use of this term in any claims. As a further example, as used in this application, the term “circuitry” also covers implementations of simply a hardware circuit or processor (or processors) or portion of a hardware circuit or processor, along with its (or their) accompanying software and / or firmware. The term circuitry also covers, for example, baseband or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices, where applicable to particular claim elements.

[0119] Those skilled in the art should understand that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the present disclosure. Similarly, it will be understood that any flowcharts, flow diagrams, state transition diagrams, pseudocode, etc. may be substantially represented on a computer-readable medium and represent various processes that may be executed by a computer or processor, whether or not a computer or processor is explicitly shown.

[0120] This Summary is intended to introduce some example embodiments, with further embodiments described in the Detailed Description and / or with reference to one or more of the drawings. This Summary is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0121] Appendix: The concept of tensor product B-splines (TPB) B-splines have the property of approximating a given curve or segment with a polynomial having a continuity constraint of a certain order. B-splines can be used in regression processes by performing data fitting on top of a B-spline model. To capture the relatively high dimensionality of multiple input variables in video processing, tensor product B-splines (TPBs) can be constructed by multiplying multiple B-spline functions. Additionally, optionally, or alternatively, TPBs, by their nature (with an inherent ability to guarantee continuity up to a certain order), can be used to provide continuous curves, thereby avoiding or significantly reducing curve-fitting operations after the initial generation of predictions, transformations, and / or mappings. For example, computationally intensive polynomial approximation algorithms that guarantee continuity across multiple polynomial pieces applied to single-channel luminance predictors can be reduced or avoided using TPB techniques.

[0122] In this section, we first introduce B-splines for a single variable (channel). We then extend this idea to multiple variables (channels) as tensor product B-splines. As an example, we explain how to use tensor product B-splines in the prediction process from SDR to EDR or from EDR to SDR.

[0123] B-spline basis functions are constructed from piecewise polynomials smoothly blended at knots to achieve the desired smoothness constraint. A B-spline basis function consists of (l+1) polynomial pieces of degree l, which are combined in a (l-1)-times continuously differentiable manner. Using a complete basis, a function f(z) can be expressed as follows via a linear combination of D=T+l-1 basis functions (T is the number of knots):

number

[0124] The zeroth order of a B-spline is defined as follows:

number

number

[0125] 8A-8D are graphs illustrating a set of B-spline basis functions of orders 0 to 3 for a uniform distribution of knot points according to an example. When using quadratic B-splines with t=8 knots, there are D=9 basis functions and 9 determining coefficients {m t In a regression scenario, for example,

number

[0126] Let's say there are P pixels in an image. Triplets

number

number

[0127] Required D y (=T+l-1) basis functions

number

number

number

number

number

[0128] The correct vector is shown below.

number

number

number

number

number

number

number

[0129]

number

number

number

number

number

number

number

number

[0130] In the previous discussion, we assumed that the location of the knot points is given. t The position of} also affects the optimization result. In other words, the optimal solution is knot point {k t} position, and Coefficient

number

[0131] Using a single-channel predictor

number

number

[0132] To improve performance, the cross-color channel predictor relies on known solutions such as MMR in Dolby Vision 1.0 to explore cross-color channel relationships. In some examples, tensor product B-splines (TPB) are applied to implement cross-color prediction. MMR, used in Dolby Vision 1.0, is a global mapping operator, while TPB can model mappings in each local partition. Therefore, TPB can outperform MMR due to its flexibility in modeling local mappings.

[0133] Take the prediction of the luma channel as an example. In each dimension, three independent basis functions are required. In this case, along the Y, C0, and C1 dimensions, D y 0, D y 1, and D y There are two basis functions. The indices of the knot points in the Y, C0, and C1 dimensions are denoted by t y , t c0 , and t cl Given a set of knot points in three dimensions, we have three individual sets of basis functions:

number

number

number

number

[0134] D y =D y 0·D y 1·D y 2, the prediction is expressed as follows:

number

number

number

number

number

number

Claims

1. generating, with an electronic encoder, a first tensor product B-spline (TPB) model representing an image, the first TPB model approximating a spatial domain coordinate reshaping function configured to shift a non-uniform distribution of local content complexity within the image to a more uniform distribution of local content complexity; generating, using the electronic encoder, a plurality of second TPB models, each of the second TPB models representing a respective patch of the image, each of the second TPB models providing an estimate of an image signal within the respective patch as a function of reshaped coordinates, the reshaped coordinates being determined using the spatial domain coordinate reshaping function; and generating, with the electronic encoder, a metadata stream including first metadata representing a set of coefficients of the first TPB model and second metadata representing sets of coefficients of the plurality of second TPB models; Including, Image compression methods.

2. The reshaped coordinates are normalized to the range [0, 1]. The image compression method according to claim 1 .

3. generating the second metadata includes compressing the set of coefficients of the plurality of second TPB models using the electronic encoder; 3. The image compression method according to claim 1.

4. The compressing step comprises: generating a set of quantized coefficients by quantizing each coefficient to a fewer number of bits; and applying entropy coding to the set of quantized coefficients; Including, The image compression method according to claim 3 .

5. the entropy coding includes arithmetic coding; the metadata stream includes side information characterizing at least one of the quantization and the arithmetic coding.

5. The image compression method according to claim 4.

6. the quantizing includes applying non-linear quantization to the coefficients of the plurality of second TPB models.

5. The image compression method according to claim 4.

7. further comprising dividing the image into a plurality of non-overlapping patches using the electronic encoder; each patch is selected from said plurality of non-overlapping patches; The image compression method according to any one of claims 1 to 6.

8. the image is a YCbCr image, The method further includes generating the YCbCr image by applying an RGB to YCbCr transform to the RGB image. The image compression method according to any one of claims 1 to 7.

9. the plurality of second TPB models include a plurality of luma channel TPB models and a plurality of chroma channel TPB models; The image compression method according to any one of claims 1 to 8.

10. The two-dimensional standard deviation is a measure of the local content complexity. The image compression method according to any one of claims 1 to 9.

11. generating the plurality of second TPB models includes finding, for a second TPB model from the plurality of second TPB models, a respective set of coefficients that approximately minimizes a difference between the image signal and an estimate of the image signal in each patch, wherein the estimate of the image signal is calculated using the second TPB model; The image compression method according to any one of claims 1 to 10.

12. said finding being based on least squares minimization; The image compression method according to claim 11.

13. transmitting the metadata stream to an electronic decoder via a communication channel. The image compression method according to any one of claims 1 to 12.

14. performing rate-distortion optimization to select one or more parameters from the group consisting of a patch size, a number of basis functions, and a number of bits for quantized TPB coefficients; The image compression method according to any one of claims 1 to 13.

15. The rate-distortion optimization is based on a greedy algorithm.

15. The image compression method according to claim 14.

16. the spatial reshaping function is a normalized cumulative function of local standard deviations calculated via a process of standard deviation equalization; The image compression method according to any one of claims 1 to 15.

17. storing instructions which, when executed by an electronic processor, cause said electronic processor to perform operations including the method of any one of claims 1 to 16; Non-transitory computer-readable medium.

18. 1. An apparatus for image compression, said apparatus comprising: at least one processor; at least one memory containing program code; Equipped with The at least one memory and the program code are used by the at least one processor to cause the device to at least: generating a first tensor product B-spline (TPB) model representing the image, the first TPB model approximating a spatial domain coordinate reshaping function configured to shift a non-uniform distribution of local content complexity within the image to a more uniform distribution of local content complexity; generating a plurality of second TPB models, each of the plurality of second TPB models representing a respective patch of the image, each of the second TPB models providing an estimate of an image signal within the respective patch as a function of reshaped coordinates, the reshaped coordinates being determined using the spatial domain coordinate reshaping function; and generating a metadata stream including first metadata representing a set of coefficients of the first TPB model and second metadata representing sets of coefficients of the plurality of second TPB models; configured to cause A device for image compression.

19. The reshaped coordinates are normalized to the range [0, 1].

20. The apparatus of claim 18.

20. receiving, with an electronic decoder, a metadata stream including first metadata representing a set of coefficients of a first tensor product B-spline (TPB) model and second metadata representing sets of coefficients of a plurality of second TPB models, the first TPB model approximating a spatial domain coordinate reshaping function configured to shift a non-uniform distribution of local content complexity within a source image to a more uniform distribution of local content complexity, each of the second TPB models representing a respective patch of the source image and providing an estimate of an image signal within each patch as a function of reshaped coordinates, the reshaped coordinates being determined based on the spatial domain coordinate reshaping function; generating, using the electronic decoder, a plurality of reconstructed image patches based on the first TPB model and further based on the plurality of second TPB models, wherein the first TPB model is reconstructed using the first metadata and the plurality of second TPB models are reconstructed using the second metadata; and constructing an estimated image by assembling the reconstructed image patches in an image frame using the electronic decoder; Including, How to decompress images.

21. The reshaped coordinates are normalized to the range [0, 1].

21. The image decompression method of claim 20.

22. and further comprising: using the electronic decoder to decompress the second metadata to reconstruct the set of coefficients of the plurality of second TPB models.

22. The image decompression method according to claim 20 or 21.

23. the decompressing is based on side information included in the metadata stream, the side information characterizing at least one of quantization and arithmetic coding for the sets of coefficients of the plurality of second TPB models.

23. The image decompression method of claim 22.

24. the plurality of second TPB models include a plurality of luma channel TPB models and a plurality of chroma channel TPB models; The image decompression method according to any one of claims 20 to 23.

25. The two-dimensional standard deviation is a measure of the local content complexity. The image decompression method according to any one of claims 20 to 24.

26. storing instructions which, when executed by an electronic processor, cause said electronic processor to perform operations including the method of any one of claims 20 to 25; Non-transitory computer-readable medium.

27. 1. An apparatus for image decompression, said apparatus comprising: at least one processor; at least one memory containing program code; Equipped with The at least one memory and the program code are used by the at least one processor to cause the device to at least: receiving a metadata stream including first metadata representing a set of coefficients of a first tensor product B-spline (TPB) model and second metadata representing sets of coefficients of a plurality of second TPB models, the first TPB model approximating a spatial domain coordinate reshaping function configured to shift a non-uniform distribution of local content complexity within a source image to a more uniform distribution of local content complexity, each of the second TPB models representing a respective patch of the source image and providing an estimate of an image signal within each patch as a function of reshaped coordinates, the reshaped coordinates being determined using the spatial domain coordinate reshaping function; generating a plurality of reconstructed image patches based on the first TPB model and further based on the plurality of second TPB models, wherein the first TPB model is reconstructed using the first metadata and the plurality of second TPB models are reconstructed using the second metadata; and constructing an estimated image by assembling the reconstructed image patches in an image frame; configured to cause Device for image decompression.

28. The reshaped coordinates are normalized to the range [0, 1].

28. The apparatus of claim 27.

Images