Generating HDR images from corresponding camera unprocessed images and SDR images

Abstract

Using the camera unprocessed image as a guide image, guide filtering is applied to the first image to generate an intermediate image. Dynamic range mapping is performed on the intermediate image to generate a second image having a different dynamic range. The second image is used to generate a particular local shaping function index value to select a particular local shaping function. A particular local shaping function is applied to the second image to generate a locally shaped image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 333,374, filed on April 21, 2022, and European Application No. 22169192.6, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to image processing operations. More specifically, embodiments of the present disclosure relate to video codecs. Background Art

[0004] As used herein, the term "dynamic range" (DR) may relate to the ability of the human visual system (HVS) to perceive a range of intensities (e.g., luminance, luma) in an image, e.g., from the darkest black (shadows) to the brightest white (highlights). In this sense, DR relates to intensities "involving a scene". DR may also relate to the ability of a display device to fully or approximately render a range of intensities of a particular width. In this sense, DR relates to intensities "involving a display". Unless a particular meaning is expressly specified at any point in the description herein to be of particular importance, it should be inferred that the term may be used in either sense, e.g., interchangeably.

[0005] As used herein, the term high dynamic range (HDR) refers to a DR width that spans 14-15 or more orders of magnitude of the human visual system (HVS). In practice, the DR of a wide width of the intensity range that humans can perceive simultaneously may be slightly truncated relative to HDR. As used herein, the terms enhanced dynamic range (EDR) or visual dynamic range (VDR) may refer alone or interchangeably to the DR that is perceptible in a scene or image by the human visual system (HVS) including eye movement, thereby allowing some light adaptation changes across scenes or images. As used herein, EDR may refer to a DR that spans 5 to 6 orders of magnitude. Although it may be slightly narrower than the HDR involved in real scenes, EDR still represents a wide DR width and may also be referred to as HDR.

[0006] In practice, an image includes one or more color components of a color space (e.g., grayscale Y and chrominance Cb and Cr), where each color component is represented with n-bit precision per pixel (e.g., n=8). Using nonlinear brightness encoding (e.g., gamma encoding), images with n≤8 (e.g., color 24-bit JPEG images) are considered as images with standard dynamic range, while images with n>8 can be considered as images with enhanced dynamic range.

[0007] A reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between the color values ​​(e.g., brightness) of an input video signal and the output screen color values ​​(e.g., screen brightness) produced by the display. For example, ITU Rec. ITU-R BT.1886, "Reference electro-optical transfer function for flat panel displays used in HDTV studio production" (March 2011), which is incorporated herein by reference in its entirety, defines a reference EOTF for flat panel displays. Given a video stream, information about its EOTF may be embedded in the bitstream as (image) metadata. The term "metadata" herein refers to any auxiliary information that is transmitted as part of the encoded bitstream and assists a decoder in rendering a decoded image. Such metadata may include, but is not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters as described herein.

[0008] The term "PQ" as used herein refers to the amplitude quantization of perceived brightness. The human visual system responds to increased light levels in a very nonlinear manner. The ability of humans to see stimuli is affected by the brightness of the stimulus, the size of the stimulus, the spatial frequencies that make up the stimulus, and the brightness level to which the eye adapts at a particular moment of viewing the stimulus. In some embodiments, the perceptual quantizer function maps a linear input grayscale to an output grayscale that better matches the contrast sensitivity threshold in the human visual system. An example PQ mapping function is described in SMPTE ST 2084:2014 "High Dynamic Range EOTF of Mastering Reference Displays" (hereinafter referred to as "SMPTE"), which is incorporated herein by reference in its entirety, where a fixed stimulus size is given, for each brightness level (e.g., stimulus level, etc.), the minimum visible contrast step size for the brightness level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to the HVS model).

[0009] Support 200cd / m 2 Up to 1000cd / m 2A display with a brightness of 0.1 or nits represents a lower dynamic range (LDR) associated with EDR (or HDR), also known as standard dynamic range (SDR). EDR content can be displayed on an EDR display that supports a higher dynamic range (e.g., from 1000 nits to 5000 nits or more). Such displays can be defined using an alternative EOTF that supports high brightness capabilities (e.g., 0 nits to 10,000 nits or more). Examples of such EOTFs are defined in SMPTE 2084 and Rec.ITU-R BT.2100 "Image parameter values ​​for high dynamic range televisionfor use in production and international programme exchange" (06 / 2017). As recognized by the inventors herein, improved techniques for generating high-quality video content data with high dynamic range, high local contrast, and vivid colors are desired.

[0010] WO 2022 / 072884 A1 discloses an adaptive local shaping method for SDR to HDR upconversion. A global index value for selecting a global shaping function for an input image with a relatively low dynamic range is generated using grayscale codewords in an input image. Image filtering is applied to the input image to generate a filtered image. The filtered value of the filtered image provides a measure of the local brightness level in the input image. Local index values ​​for selecting a specific local shaping function for the input image are generated using the global index value and the filtered value of the filtered image. By shaping the input image with a specific local shaping function selected using the local index value, a shaped image with a relatively high dynamic range is generated.

[0011] CN 112 200 719A discloses an image processing method, comprising the following steps: obtaining a first resolution standard dynamic range SDR image; obtaining a guide map of a second resolution SDR image and a filter coefficient of the second resolution SDR image according to the first resolution SDR image; and performing guided filtering processing according to the guide map of the second resolution SDR image and the filter coefficient of the second resolution SDR image to obtain a second resolution HDR image, wherein the second resolution is higher than the first resolution. According to the present disclosure, the calculation complexity of the filter coefficient can be reduced, the fast mapping of the SDR image to the HDR image can be realized, and the problem of large calculation amount of the mapping method of the SDR image to the HDR image can be solved.

[0012] WO 2020 / 210472 A1 discloses a method for generating a high dynamic range (HDR) image. The method comprises: (a) denoising a short exposure time image, wherein the denoising comprises applying a first guided filter to the short exposure time image, the guided filter using the long exposure time image as its guide; (b) after the denoising step, scaling at least one of the short exposure time image and the long exposure time image to place the short exposure time image and the long exposure time image on a common radiation scale; and (c) after the scaling step, merging the short exposure time image with the long exposure time image to generate an HDR image.

[0013] US2015 / 078661 A1 discloses an algorithm for improving the performance of a conventional tone mapping operator (TMO) by calculating a contrast waste fraction and a contrast loss fraction of a first tone mapped image produced by the TMO. These two contrast scores can be used to optimize the performance of the TMO by reducing noise and improving contrast. An algorithm for generating an HDR image by converting a non-linear color space image into a linear color space format, aligning the image with a reference, deghosting the aligned image if necessary, and merging the aligned (and possibly deghosted) images to create an HDR image. The merging can be performed using exposure fusion, HDR reconstruction, or other suitable techniques.

[0014] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any approach described in this section qualifies as prior art simply by virtue of its inclusion in this section. Similarly, unless otherwise indicated, it should not be assumed that the problems identified with respect to one or more approaches based on this section have been recognized in any prior art. Summary of the invention

[0015] The invention is defined by the independent claims. The dependent claims are directed to optional features of some embodiments of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Embodiments of the invention are illustrated by way of example and not limitation in the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0017] Figure 1 illustrates an example conversion from a camera raw image and an SDR image to a corresponding HDR image;

[0018] Figure 2A illustrates an example generation of an enhanced SDR image as an intermediate image; Figure 2B illustrates an example construction of a static three-dimensional lookup table (3D-LUT); Figure 2C An example image enhancement is illustrated;

[0019] Figure 3A illustrates an example compressed and filtered SDR image in guided filtering with a relatively small kernel; Figure 3B and Figure 3C Example codeword graphs for JPEG SDR, filtered SDR, and / or camera raw images are illustrated; Figure 3D illustrates an example compressed and filtered SDR image in guided filtering with a relatively large kernel; Figure 3E An example two-dimensional plot of local standard deviation with smoothed binarization is illustrated; Figure 3F illustrates an example compressed and fused SDR image; Figure 3G illustrates example mapped HDR images generated using a static 3D-LUT with and without darkness adjustment post-processing; Figure 3H An example curve for generating a hybrid shifted sigmoid lookup table (HSSL LUT) is shown; Fig. 3I and Figure 3J An example local shaping function is illustrated;

[0020] Figure 4A and Figure 4B illustrates an example process flow; and

[0021] Figure 5 Illustrated is a simplified block diagram of an example hardware platform upon which a computer or computing device as described herein may be implemented. DETAILED DESCRIPTION

[0022] In the following description, for ease of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it is clear that the present disclosure can be practiced without these specific details. In other cases, well-known structures and devices are not described in detail to avoid unnecessarily obscuring, obscuring or confusing the present disclosure.

[0023] Overview

[0024] An image acquisition device may be used to capture a camera raw image. A camera raw image stores raw (physical or optical) sensor information (or sensor readout) that has not yet been processed by a camera image signal processor (ISP). An image signal processing operation may be performed on the camera raw image by the camera ISP of the image acquisition device to generate a (JPEG) SDR image (such as a thumbnail image with a relatively small spatial resolution and a relatively low dynamic range) corresponding to the camera raw image. The camera raw image and the SDR image may be included in a single (raw) image container or file or packaged digitally. Examples of such image containers may include, but are not necessarily limited to, any of the following: a DNG (digital negative) file, a TIFF file, a BMP file, a JPEG file, and the like.

[0025] According to the techniques described herein, camera raw images and SDR images can be converted into corresponding HDR images of relatively high quality with a visually pleasing appearance that is comparable to or better than the HDR images generated by the device. These techniques can be implemented by the image acquisition device itself or by a separate computing device as part of an image processing tool outside the ISP. A non-limiting example of such an image processing tool outside the ISP can be, but is not necessarily limited to, Adobe Photoshop, which a user can operate to create (one or more) images that they prefer using the camera raw image as a starting point.

[0026] In many operating scenarios, although the JPEG SDR image has a relatively beautiful picture appearance, it is generated by using the camera ISP of the image acquisition device (e.g., dedicated, etc.) to perform a large number of local mapping / processing operations on different spatial regions or image blocks on the camera raw image, with a relatively low spatial resolution and / or a relatively low bit depth compared to the camera raw image.

[0027] In contrast, while not necessarily visually pleasing, a Camera Raw image does contain relatively rich image data—such as image detail at the highest resolution supported by the image sensor used to generate the Camera Raw image.

[0028] According to the techniques described herein, by making full use of both the SDR image and the camera raw image, a relatively high quality (or visually pleasing) HDR image can be generated from an SDR image having a large amount of details from the camera raw image. The camera raw image can be used to help create an HDR image with better image details than the SDR image. At the same time, given that the SDR image has a relatively pleasing picture appearance (or SDR picture mode) supported by the image acquisition device or camera, the SDR image can be used together with the camera raw image to help provide or present the best visual appearance in the resulting HDR image.

[0029] In some operating scenarios, the framework for implementing these techniques may include multiple (e.g., three, etc.) stages or main components. The first stage uses local linear mapping and / or guided filtering with different (spatial filtering) kernel sizes to generate filtered SDR images from camera raw data and (e.g., JPEG, etc.) SDR images and generate enhanced SDR images by fusing the filtered SDR images. The second stage converts the enhanced SDR images into HDR images using static SDR to HDR mapping. Static mapping can be defined or specified using operating parameters learned or optimized in a training process using training RAW / SDR / HDR images. The training images used to train the optimized operating parameters of static SDR to HDR mapping may include, but are not limited to, spatially aligned SDR and HDR images generated from captured SDR and HDR images or pictures. The third stage applies local shaping or (one or more) mixed shifted S-shaped functions for image enhancement to further enrich the contrast, image details, color saturation, etc. in the HDR image generated in the second stage. It should be noted that the local linear mapping tool used in the first stage and the image enhancement tool used in the third stage can be applied to different image processing applications or operation scenarios, and are not necessarily limited to converting RAW / SDR images into corresponding HDR images as described herein. For example, some or all of these local linear mapping and / or image enhancement tools can be used alone or in combination to provide a relatively rich set of operating parameters for users to select and customize the converted HDR image to have different preferred visual appearances.

[0030] In some operating scenarios, training camera raw / SDR / HDR images may be obtained or captured with a mobile device. One or more such mobile devices may be operated to take or capture (training) camera raw and SDR images and (training) HDR images, such as in a captured Profile 8.4 video. The collected / captured training camera raw / SDR images and the collected / captured training HDR images may be used to establish or specify training mappings or pairs, each mapping or pair including a training camera raw image, a training SDR image corresponding to the training camera raw image, and a training HDR image corresponding to the training camera raw image and the training SDR image. These training images may then be used to optimize operating parameters in an image processing or enhancement tool as described herein to generate optimized operating parameters.

[0031] The multi-stage framework as described herein can be deployed to or adapted to different image acquisition devices or cameras with similar training processes. Once these image processing tools including prediction / mapping models or optimized operating parameters for each stage of the framework are generated or trained through (one or more) training processes as described herein, these image processing tools or modules can be deployed, installed, downloaded and / or executed on the mobile device side or cloud-based server side.

[0032] Example embodiments described herein relate to generating an image of a first dynamic range from an input image of a second dynamic range different from the first dynamic range. A guided filter is applied to the first image using a camera raw image as a guide image to generate an intermediate image. A first image of the first dynamic range is generated from the camera raw image using an image signal processor of an image acquisition device. Dynamic range mapping is performed on the intermediate image of the first dynamic range to generate a second image of a second dynamic range different from the first dynamic range. The second image of the second dynamic range is used to generate a specific local shaping function index value for selecting a specific local shaping function for the second image of the second dynamic range. The specific local shaping function is applied to the second image of the second dynamic range to generate a locally shaped image of the second dynamic range.

[0033] Example Video Delivery Processing Pipeline

[0034] Figure 1 An example conversion framework for converting a camera raw image (along with an SDR image) into a corresponding HDR image is illustrated. By way of illustration and not limitation, the conversion framework includes three main components or processing blocks. Some or all of the conversion framework or the components / processing blocks therein may be implemented in software, hardware, a combination of software and hardware, etc.

[0035] The first major component or processing block of the conversion framework is used to receive the camera raw image and the SDR image, and process the received image to generate an intermediate image. The received SDR image can be generated by the camera ISP of the image acquisition device, while the camera of the image acquisition device simultaneously captures or acquires the camera raw image. The received SDR image can be used as a relatively low resolution, low dynamic range image or thumbnail image representing the relatively high resolution, relatively high dynamic range camera raw image. The example intermediate image can be, but is not necessarily limited to, an enhanced SDR image.

[0036] In some operational scenarios, the camera raw image and the SDR image may be digitally packaged or wrapped together in an input image container or file received as input by a first major component or processing block. Such an image container or file may contain a relatively rich collection of image data, including but not limited to a camera raw image in the form of a raw raw sensor image, an SDR image in the form of a JPEG SDR thumbnail image, and the like. The camera raw image represents light captured by a light sensor, while the SDR image represents a processed (e.g., SDR, thumbnail, preview, etc.) image appearance generated by a camera ISP performing a relatively large amount of global / local image processing on the camera raw image. The techniques described herein may utilize both the camera raw image and the SDR image to generate an intermediate image (such as an enhanced SDR image) for use by the next major component or processing block (or stage) or to perform further image processing operations.

[0037] The second main component or processing block is used to receive the enhanced image generated by the first main component or processing block as input and convert it into an HDR image using one of various types of dynamic and / or static (image conversion / mapping) models. As an illustration and not an example, the static model can be implemented as a static three-dimensional lookup table (3D-LUT) and constructed from a mapping / function / model based on tensor product B-spline (TPB) in the RGB domain, which has a relatively low computational complexity or cost. In addition, optionally or alternatively, the 3D-LUT as described herein can be constructed from a three-dimensional mapping table (3DMT). A 3D-LUT constructed with a 3DMT may cause more irregularities and discontinuities, and is therefore more likely to cause stripes or color artifacts. In contrast, a 3D-LUT constructed with a TPB-based model may have fewer irregularities and smoother continuity in nearby codeword nodes or entries of the 3D-LUT or in a local neighborhood of the codeword space, thereby providing a more perceptually pleasing appearance. Example TPB-based mappings / functions / models can be found in U.S. Provisional Patent Application No. 62 / 908,770, filed on October 1, 2019 by Guan-Ming Su, Harshad Kadu, Qing Song, and Neeraj J. Gadgil, entitled “Tensor-product B-spline predictor,” the contents of which are incorporated herein by reference in their entirety as if fully set forth herein. Example operations for building a lookup table (such as a 3D-LUT) from a 3DMT for SDR to HDR mapping or conversion can be found in U.S. Patent Application No. 17 / 054,495, filed on May 9, 2019, the contents of which are incorporated herein by reference in their entirety as if fully set forth herein.

[0038] The third main component or processing block in the framework is used to receive the (mapped or converted) HDR image generated by the second main component or processing block as input, and enhance or improve the local contrast, local image detail and local saturation in the mapped HDR image to generate an output HDR image. Similar to the SDR image output from the camera ISP, the HDR image output from the camera ISP is also enhanced by a local image processing tool used by or with the camera ISP to give it a visually pleasing appearance. The framework as described herein or the third main component or processing block therein can be used to support or provide a visually pleasing appearance of the same or similar level as the native HDR image output by the camera ISP output HDR image. In some operating scenarios, the framework can be used to provide an HDR image enhancement component that enhances the (mapped or converted) HDR image to make it appear perceptually or visually attractive, for example, outside or replacing the camera ISP or a local image processing tool that operates in conjunction with the camera ISP.

[0039] Raw image processing

[0040] Figure 2A Picture shows Figure 1 The first major component or processing block ("raw image processing") used to generate an enhanced SDR image as an example of an intermediate image. Figure 2A As shown in , resolution and bit depth adjustment can be performed on a (JPEG) SDR image and a camera raw image corresponding to or generating the (JPEG) SDR image, respectively. A first guided filter with a relatively small spatial kernel can be run on the adjusted SDR image using enhanced information extracted from the adjusted camera raw image to generate a first filtered SDR image. A second guided filter with a relatively large spatial kernel can also be run on the adjusted SDR image using enhanced information extracted from the adjusted camera raw image to generate a second filtered SDR image. A local standard deviation can be determined in the adjusted (JPEG) SDR image and used to derive a fusion weighting factor. Image fusion can be performed on the first and second filtered SDR images based on the (fusion) weighting factor to generate an enhanced SDR image.

[0041] As described above, the raw image or the image container / file storing the camera raw image may also include an image generated by the camera ISP, such as an SDR image. In many operating scenarios, the (e.g., JPEG, etc.) SDR image generated by the camera ISP from the camera raw image already provides a relatively high visual quality (appearance) because the camera ISP has performed many image processing operations, such as local tone mapping, noise reduction, detail enhancement, color enrichment, etc. In order to generate a visually pleasing HDR image corresponding to or based on the image data stored in the raw image or image container / file, the (JPEG) SDR image may be used as a (e.g., good, etc.) starting point. The (JPEG) SDR may discard some details or suffer from compression artifacts during (camera) ISP processing. Under an image fusion framework implemented by techniques as described herein, some or all of the detailed information missing from an SDR image resulting from ISP processing or compression may be retrieved or re-retrieved from sensor raw data stored in a camera raw image, which sensor raw data may have a higher spatial resolution and contain all image data, information, and / or details prior to all image processing operations used to derive a (JPEG or compressed) SDR image. Under such an image fusion framework, the image data, information, and / or details extracted from the camera raw image may be used as additional information to be merged or fused together with the image data, information, and / or details represented or contained in the (JPEG) SDR image to generate an intermediate image. The intermediate image may be an enhanced SDR image compared to the (JPEG) SDR image, and may be used to generate a relatively high quality HDR image by applying an SDR to HDR conversion to the enhanced SDR image instead of the (JPEG) SDR image.

[0042] The camera raw (or sensor) image and the (JPEG) SDR image may have different spatial resolutions, and the camera raw image often has a higher spatial resolution than the (JPEG) SDR image. The camera raw (or sensor) image and the (JPEG) SDR image may have different bit depths. In order to combine or fuse the image data or information in the camera raw image and the SDR image, the SDR image may be upsampled to the same resolution as the camera raw image. Where applicable, one or both of the camera raw image and the SDR image may be converted to a common bit depth (such as 16 bits), or each codeword or codeword component in the image may be represented as a 16-bit value. Through these adjustment operations, the spatial resolution and bit depth of the (adjusted) camera raw image and the SDR image are aligned.

[0043] Denote the i-th pixel of the (adjusted) camera raw (or sensor) image as r i , denote the i-th pixel of the corresponding (adjusted JPEG) SDR image as s i .

[0044] Several image enhancement tasks may be performed to generate an enhanced SDR image as an intermediate image. First, some or all missing or lost information in the (JPEG) SDR image may be restored (due to or attributable to relatively small spatial resolution, quantization distortion associated with relatively low bit depth (e.g., 8-bit JPEG, etc.), image compression artifacts due to lossy compression when generating JPEG SDR). Second, some or all clipped information (or clipped codeword values) in the (JPEG) SDR image may also be restored, which is caused by tone mapping performed by the camera ISP to fit the (specified) SDR dynamic range.

[0045] Artifacts may arise due to the spatial resolution resampling operation, especially near the edges that separate different image details (e.g., the outline of a human body, the outline of an eye, etc.). Since the previously downsampled JPEG-SDR image (e.g., as part of the camera ISP processing, etc.) is upsampled back to a higher spatial resolution, down to the resolution of the camera raw image, these edges that separate image details will be blurred and will show visual distortions, such as jagged "pixelation" artifacts. As part of the image enhancement task, these (edge) artifacts can be considered as noise in the SDR image. The original or relatively accurate edge information in the camera raw image can be extracted and used with edge-preserving filtering to enhance the SDR image including the distorted edges.

[0046] In some operating scenarios, the guided filter can be used as an edge-preserving filter to perform edge-preserving filtering relatively efficiently. Since the edge is visually shown or depicted in relatively small image details or relatively small spatial scales / regions, the guided filter can be set or configured to have a relatively small kernel size.

[0047] Artifacts may also be produced due to tone mapping used by the camera ISP to map the camera raw image to the SDR image, especially in highlight and dark areas caused by (codeword value range) clipping. This occurs when the camera ISP adjusts the input (e.g., relatively high, etc.) dynamic range in all codewords or pixel values ​​across the entire captured scene to an output (e.g., relatively low, etc.) dynamic range (e.g., SDR). The camera ISP may perform a tradeoff to keep or maintain more codewords in the mid-tone areas of the captured scene at the expense of using fewer codewords or clipping in the highlight or dark areas of the captured scene.

[0048] It is assumed that the local tone mapping of a local block (e.g., a specific image detail, a specific image coding unit, a specific image block, etc.) of a camera raw image to a corresponding local block of a (JPEG) SDR image is linear. Local image data or information in the corresponding local blocks of the camera raw image and the (JPEG) SDR image can be collected to construct or reconstruct a linear model for local tone mapping. This "local" mapping can be accomplished by moving a sliding window through each of a plurality of local neighborhoods that constitute some or all of the (JPEG) SDR image. A second guided filter can be used to help construct the local mapping. Since the highlights or dark areas through which this local tone mapping attempts to recover the missing clipped codewords or pixel values ​​are relatively large compared to the edges that depict or separate different image details, a relatively large kernel size can be used with the second guided filter.

[0049] For the guided filter, assume that in the (adjusted JPEG) SDR image as the input image (where the i-th pixel can be represented as s i , as mentioned earlier) and the adjusted camera raw image as the guide image (where the i-th pixel can be represented as r i , as mentioned earlier). Given a local neighborhood of size BxB (or B pixel rows by B pixel columns) (denoted as Φ i ), centered on the ith pixel in each of the input image and the guidance image, the local linear correlation in the neighborhood can be expressed as a linear function of the input and guidance images (α i ,β i ) as follows:

[0050]

[0051] Input image(s i ) and the predicted image (where the i-th pixel is represented as ) can be modeled or represented as noise or a noise image (where the i-th pixel is represented as n i ), as follows:

[0052]

[0053] A linear function used to model local linear dependencies, or an operating parameter such as the slope α i and the offset β i ), can be obtained by minimizing s i and The difference between is determined or found as follows:

[0054]

[0055]

[0056] where ∈(α i ) 2 represents a regularization term of a relatively small value used to ensure convergence to the optimization / minimization solution.

[0057] The optimized solution can be obtained or calculated as follows:

[0058]

[0059] where μ i and (σ i ) 2 Represents Φ i Medium i The mean and variance of .

[0060] To avoid abrupt changes or discontinuities, the linear function used to model the local correlation in the local neighborhood between the input image and the guidance image may utilize the average of the optimized operating parameters calculated for some or all pixels in the local neighborhood of the i-th pixel (denoted as and ) to ensure smoothness between adjacent pixels, as shown below:

[0061]

[0062] Where k represents the neighborhood Φ of the i-th pixel i Pixels in .

[0063] The final (guided filtered or predicted) value of the ith pixel can be given as:

[0064]

[0065] The guided filter operation shown in the above expressions (1) to (6) can be expressed as follows:

[0066]

[0067] Spatial detail enhancement based on resampling

[0068] In order to reconstruct image or spatial details (including but not limited to details related to edges lost by resampling (or downsampling and / or upsampling)), the guided filter in the above expression (7) can be deployed or applied to a relatively small neighborhood with a relatively small B value. The relatively small B value for recovering image or spatial details lost due to resampling can be expressed as B S. B S Example values ​​of can be, but are not necessarily limited to, any one of 12, 15, 18, etc.

[0069] With this relatively small value B S The deployment or application of the guided filter can be expressed as follows:

[0070]

[0071] Figure 3A An example (JPEG) SDR image is illustrated on the left, and an example filtered SDR image is illustrated on the right, wherein a guided filter with a relatively small neighborhood (or spatial kernel) is combined with enhanced information from a camera raw image corresponding to or generating the (JPEG) SDR image. As shown, a large amount of detail is recovered via the guided filter with a relatively small neighborhood (or spatial kernel) with the aid of enhanced information from the camera raw image.

[0072] Figure 3B Three example graphs of codeword values ​​in corresponding rows of a (JPEG) SDR image, a filtered SDR image, and a camera raw image are illustrated. More specifically, the first graph depicts the compressed codeword values ​​for a row of the (JPEG) SDR image, the second graph depicts the filtered codeword values ​​for a row of the filtered SDR image (corresponding to the row of the (JPEG) SDR image), and the third graph depicts the camera raw codeword values ​​for a row of the camera raw image (corresponding to the row of the (JPEG) SDR image). For ease of comparison, Figure 3C The first and second figures are superimposed together in FIG. As shown, the guided filter increases the total number of codeword values ​​in the filtered codeword values ​​of the filtered SDR image rows compared to the compressed codeword values ​​of the (JPEG) SDR image rows, thereby causing the filtered SDR image to contain more image details represented in the camera raw image. Therefore, the guided filter effectively increases the bit depth of the filtered SDR image and significantly mitigates banding artifacts that may otherwise be caused by the relatively small total number of compressed codewords used in the (JPEG) SDR image.

[0073] Spatial detail enhancement based on clip

[0074] As described above, assuming that the mapping implemented by the camera ISP to convert the camera raw image into a (JPEG) SDR mapping is locally linear, then a (second) guided filter can be deployed or applied to recover spatial details lost due to (codeword) clipping, such as in highlights or dark areas. Spatial regions affected by clipping in (JPEG) SDR images typically appear within or above large spatial blocks. Since pixels within the spatial region have constant values ​​(e.g., including the upper or lower bounds of the codeword space of available codewords for encoding, etc.), a small kernel size may not be able to recover the details lost due to clipping, and instead may produce a linear function that is trained to map different non-constant (codeword or pixel) values ​​to constant (codeword or pixel) values. Such a linear function is ineffective for recovering details lost due to clipping. Therefore, a relatively large neighborhood or kernel size can be used with the (second) guided filter to cover a relatively large block including non-clipped pixels and clipped pixels. Accordingly, a relatively large value can be selected for the parameter B in the above expression (7), denoted as B L . B L Example values ​​of can be, but are not necessarily limited to, any of the following: 512, 1024, 2048, etc.

[0075] With this relatively large value B L The deployment or application of the guided filter can be expressed as follows:

[0076]

[0077] Figure 3D An example (JPEG) SDR image is illustrated on the left and an example filtered SDR image is illustrated on the right, wherein a relatively large neighborhood (or spatial kernel) guided filter combines enhanced information from a camera raw image corresponding to or generating the (JPEG) SDR image. As shown, visually noticeable spatial details lost in the (JPEG) SDR image due to clipping (such as clouds depicted in the image) have been restored in the filtered SDR image by using the enhanced information from the camera raw image via the (second) guided filter, although some colors and details depicted in the (JPEG) SDR image have been altered in non-clipped spatial regions or blocks (such as buildings and trees) in the filtered SDR image.

[0078] In some operating scenarios, the relatively large kernel size used in the guided filtering can be set equal to or even larger than the entire picture / frame. This is equivalent to having a global mapping from the camera raw image to the (JPEG) SDR image. While this model can be implemented with relatively high computational efficiency and relatively low computational complexity, the resulting filtered SDR image may not contain as much clipped spatial and / or color detail in the previously clipped spatial region / block compared to a filtered SDR image generated using a relatively large but smaller kernel size than the entire picture / frame. In this case, we can derive a simple and very fast global linear regression model.

[0079] Image fusion of multiple enhanced or filtered images

[0080] Although guided filters can be used to reconstruct lost details, filtered images generated from guided filters or filters with different kernel sizes have different advantages and disadvantages. For example, a guided filter with a relatively small kernel size may not be able to recover spatial or color details from clipping, while a guided filter with a relatively large kernel size may affect color or brightness and even blur image details in unclipped spatial regions / blocks.

[0081] In order to take advantage of both small and large kernel sizes, image fusion can be used to combine different filtered images into an overall filtered image. Since a guided filter with a relatively large kernel size will increase or restore missing spatial and / or color details in a cropped region of a JPEG SDR image, the guided filter can be limited to being applied to the cropped spatial region / region. In some operating scenarios, a cropped spatial region / region is detected in a (JPEG) SDR image. A weighted factor map used to achieve image fusion can be generated to limit or prevent the second guided filter from adversely affecting the uncut spatial region / region.

[0082] Figure 4A An example process flow for generating a weighting factor map for a combination of a first guided filter having a relatively small kernel size and a second guided filter having a relatively large kernel size is illustrated. This process flow can be run repeatedly to generate a specific weighting factor map for each color channel of a color space in which a JPEG SDR image and / or a filtered image is represented.

[0083] Block 402 includes calculating the local standard deviation. For each (eg, i-th, etc.) SDR pixel in the (JPEG) SDR image, a standard deviation filter may be used to calculate the local standard deviation represented by d i The local standard deviation of the filter kernel is set to B centered on or around the i-th SDR pixel. dxB d Pixel neighborhood ψ i , as shown below:

[0084]

[0085] B d A non-limiting example value of may be, but is not necessarily limited to, 9. The local standard deviation may be represented as a two-dimensional local standard deviation map, where each pixel has a pixel value that is the local standard deviation calculated for that pixel.

[0086] Block 404 includes binarizing the local standard deviation output from the standard deviation filter. d The local standard deviation map or the local standard deviation therein is binarized into a binary mask as follows:

[0087]

[0088] Threshold T d A non-limiting example value of may be, but is not necessarily limited to, 0.01.

[0089] Block 406 includes applying a Gaussian spatial filter to construct smooth spatial transitions in the spatial distribution of binary values ​​0 and 1 in the binary mask to generate a local standard deviation of the binarization of the spatially smooth transition, denoted as The Gaussian spatial filter can be given as:

[0090]

[0091] where σ G Represents standard deviation. G Example values ​​of may include, but are not necessarily limited to, one of 32, 64, 128, etc.

[0092] The local standard deviation of the binarization with smooth spatial transition can be represented as a two-dimensional graph, such as Figure 3E As shown in , where each pixel has a pixel value that is the local standard deviation of the binarization of the spatially smooth transition for that pixel.

[0093] Block 408 includes generating and normalizing or renormalizing weighting factors for combining filtered images generated by guided filters having relatively small and large spatial kernel sizes.

[0094] The local standard deviation of the binarization that can smoothly transition over all spaces Determine or find the maximum and minimum values ​​(represented by and ). Maximum and minimum values and The local standard deviation can be used to binarize the spatially smooth transition Normalization, for example, normalization to a normalization value range [0 1].

[0095] A representation of the filtered pixel or codeword values ​​to be applied to a first filtered SDR image generated by a first guided filter with a relatively small spatial kernel is The first weighting factor can be set to the local standard deviation of the normalized spatially smoothed binarization as follows:

[0096]

[0097] A representation of the filtered pixel or codeword values ​​to be applied to a second filtered SDR image generated by a second guided filter with a relatively large spatial kernel is The second weighting factor can be set as the complementary value of the first weighting factor as follows:

[0098]

[0099] Block 410 includes using first and second weighting factors and For the first filtered SDR image and the second filtered SDR image Perform image fusion as follows:

[0100]

[0101] Figure 3F The left side illustrates an example (JPEG) SDR image with cropped and resampled codewords or pixel values, and an example enhanced or fused SDR image generated by fusing guided filtered SDR images generated by guided filtering using relatively small and large spatial kernels. This enhanced or fused (guided filtered) SDR image can be used as an intermediate image and provided to Figure 1 The second major component or processing block (“Enhanced SDR to HDR Conversion”) is used as part of the input to construct an HDR image corresponding to the (JPEG) SDR image and / or the camera raw image.

[0102] Enhanced SDR to HDR conversion

[0103] Figure 1 The second major component or processing block (“Enhanced SDR to HDR Conversion”) starts with Figure 1The first major component or processing block ("raw image processing") receives the enhanced SDR image and converts or maps (each SDR pixel or codeword value of) the enhanced SDR image to (the corresponding HDR pixel or codeword value of) an HDR image corresponding to the (JPEG) SDR image and the camera raw image used to generate the enhanced SDR image. In some operating scenarios, the SDR to HDR conversion or mapping can be performed via a static 3D-LUT, for example, the static 3D-LUT can map R.709RGB SDR pixel or codeword values ​​as input to R.2020RGB HDR pixel or codeword values ​​as output.

[0104] Figure 2B An example construction of a static 3D-LUT as described herein is illustrated. For example, the construction of a 3D-LUT may include three stages, such as building a TPB SDR to HDR conversion / mapping model, constructing a static 3D-LUT using the TPB model, performing 3D-LUT post-processing operations (such as node modification operations, etc.).

[0105] The TPB model can be constructed with TPB basis functions that ensure smooth transitions and continuity, thereby eliminating or avoiding discontinuities within (one or more) local neighborhoods, which can cause sudden color change artifacts in the converted / mapped HDR image. Applying the TPB model by directly using TPB coefficients and TPB basis functions in image conversion operations can cause relatively high computational complexity or cost. In some operating scenarios, a computationally efficient (e.g., static, etc.) 3D-LUT can be constructed or pre-constructed from the TPB model. The 3D-LUT can be used at runtime to map or convert images instead of directly using the TPB coefficients and TPB basis functions of the TPB model at runtime. In addition, optionally or alternatively, the 3D-LUT can be refined, for example, by modifying the 3D-LUT in a post-processing stage to refine it into a final static 3D-LUT that can be used to enhance the dark areas in the mapped or converted HDR image.

[0106] By way of illustration and not limitation, the same (physical) scene may be captured with two image acquisition devices of the same type and / or model and / or manufacturer to generate two image data sets.

[0107] For example, one of the two image acquisition devices may be operated to generate a plurality of raw image files / containers (e.g., DNG still images or image files, etc.), the plurality of raw image files / containers including a plurality of (device-generated) camera raw images and a plurality of (device-generated JPEG) SDR images. Each of the plurality of raw image files / containers includes a camera raw image and a (JPEG) SDR image generated from the camera raw image by camera ISP processing of the image acquisition device.

[0108] At or about the same time, the other of the two image acquisition devices may be operated to generate an HDR video (e.g., Profile 8.4 video, etc.) comprising a plurality or a series of (device-generated) HDR images corresponding to or depicting the same scene as the plurality of camera raw images and the plurality of (JPEG) SDR images. Each of the plurality of HDR images corresponds to or depicts the same scene as a corresponding camera raw image in the plurality of camera raw images and a corresponding (JPEG) SDR image in the plurality of (JPEG) SDR images.

[0109] Multiple HDR images, multiple camera images, and multiple (JPEG) SDR images can be used or collected as an image data set including multiple image combinations. Each image combination in the multiple image combinations includes an HDR image in the multiple HDR images, a camera raw image corresponding to the HDR image, and a (JPEG) SDR image generated from the camera raw image by camera ISP processing.

[0110] In some operating scenarios, some or all of the images in the image combination can be freely acquired or captured by two image acquisition devices without any spatial or motion constraints. In some operating scenarios, some or all of the image combinations can be acquired by two image acquisition devices under spatial or motion constraints, such as by tripod implementation or setup.

[0111] Since the multiple device-generated (JPEG) SDR images and the multiple device-generated HDR images in the image data set are acquired using two image acquisition devices, the device-generated (JPEG) SDR images in the multiple device-generated (JPEG) SDR images may not be spatially aligned with the corresponding device-generated HDR images in the multiple device-generated HDR images. Feature points can be extracted from the multiple device-generated (JPEG) SDR images and the multiple device-generated HDR images, and used to perform a spatial alignment operation on the multiple device-generated (JPEG) SDR images and the multiple device-generated HDR images to generate multiple spatially aligned device-generated (JPEG) SDR images and multiple spatially aligned device-generated HDR images. Some or all of the SDR pixels in each of the multiple spatially aligned device-generated (JPEG) SDR images are spatially aligned with some or all of the corresponding HDR pixels in the corresponding spatially aligned device-generated HDR images in the multiple spatially aligned device-generated HDR images.

[0112] Multiple spatially aligned device-generated (JPEG) SDR images and multiple spatially aligned device-generated HDR images can be used as training SDR and HDR images to train or optimize operating parameters of a specified or defined TPB conversion / mapping model that predicts or outputs predicted, mapped or converted HDR pixel or codeword values ​​based on SDR pixel or codeword values ​​as input.

[0113] For example, the SDR pixel or codeword value in each of a plurality of spatially aligned device-generated (JPEG) SDR images can be used by the TPB model to predict a mapped / converted HDR pixel or codeword value as an output. The prediction error can be measured or generated as the difference between the (predicted / mapped / converted) HDR pixel or codeword value and the corresponding HDR pixel or codeword value in the corresponding spatially aligned device-generated HR image in a plurality of spatially aligned device-generated HDR images in the image data set. The operating parameters for the TPB model can be optimized by minimizing the prediction error during the model training phase. The TPB model with optimized parameters can be used to predict the converted / mapped HDR pixel or codeword value based on the SDR pixel or codeword value.

[0114] Example feature point extraction, image spatial alignment, and TPB model training can be found in U.S. Provisional Patent Application Serial No. 63 / 321,390, filed by Guan-Ming Su et al. on March 18, 2022, entitled “Image Optimization in Mobile Capture and Editing Applications,” the contents of which are incorporated herein by reference in their entirety as if fully set forth herein.

[0115] The optimized operating parameters for the TPB model (or reverse shaping) may include those represented as The TPB coefficients of , where ch = red or R, green or G, and blue or B, are in the RGB domain or color space in which SDR and HDR images are represented. These TPB coefficients can be used in conjunction with the TPB basis functions in the TPB model to generate predicted / converted / mapped HDR pixel or codeword values ​​from SDR pixel or codeword values.

[0116] As described above, a TPB model with optimized TPB coefficients and basis functions can be used directly to predict HDR values ​​from SDR values ​​with relatively large computational complexity and cost. In some operating scenarios, in order to reduce runtime computational load, the TPB model may not be directly deployed at runtime to predict HDR values ​​in a mapped / converted HDR image from (non-trained) SDR values ​​in an SDR image. A static 3D-LUT can be constructed from a TPB model with optimized TPB coefficients and basis functions before runtime. Each of the three dimensions of the 3D-LUT is used to predict the component HDR codeword values ​​in the corresponding dimension or color channel of the HDR color space in which the output will represent the mapped / converted HDR image.

[0117] By way of illustration but not limitation, each dimension or color channel of an input SDR color space (such as an RGB color space) representing an SDR image may be partitioned into N (e.g., equal, unequal, etc.) partitions, intervals, or nodes, where N is a positive integer greater than one (1). Example values ​​of N may include, but are not limited to, any of the following values: 17, 33, 65, etc. These N partitions, intervals, or nodes in each such dimension or color channel may be represented as N corresponding component SDR codeword values ​​in the dimension or color channel. Each of the N partitions, intervals, or nodes of the dimension or color channel in the N partitions, intervals, or nodes may be represented as a corresponding component SDR codeword value in the N corresponding component SDR codeword values.

[0118] The static 3D-LUT includes NxNxN entries. Each entry of the 3D-LUT contains a different combination of three component SDR codeword values ​​as a lookup key. Each of the three component SDR codeword values ​​is located in the N component SDR codeword values ​​in one of the dimensions or color channels of the input SDR color space. Each entry has or outputs three mapped / converted component HDR codeword values ​​(as predicted by the TPB model) in each of the three dimensions or color channels of the output HDR color space as a lookup value.

[0119] Given an (input) SDR codeword value comprising three component SDR codeword values ​​(e.g., for each pixel of the input SDR image, for the i-th pixel of the input SDR image, etc.), the nearest neighboring partitions, intervals or nodes represented by the corresponding component SDR codeword values ​​of the component SDR codeword values ​​closest to the input SDR codeword value (e.g., generated by quantizing or dividing the entire codeword range (such as the normalized value range [0,1]) using the total number of partitions, intervals or nodes in each color channel, etc.) can be identified in the partitions, intervals or nodes represented in the static 3D-LUT. Each nearest neighboring partition, interval or node can be identified or indexed by a node index (a, b, c), where 0≤a, b, c≤N and corresponds to a specific entry in the 3D-LUT. The predicted component HDR codeword values ​​in channel ch can be output as part / component / dimension of the lookup value in the specific entry of the 3D-LUT using the node index identified by or converted from the component SDR codeword values ​​of the input SDR codeword value. The predicted component HDR codeword values ​​in a particular channel of the output HDR color space denoted as ch may be used from some or all entries (of the 3D-LUT) corresponding to the (e.g., two, four, six, eight, etc.) nearest neighboring partitions, bins, or nodes - e.g., by bilinear interpolation, trilinear interpolation, extrapolation, etc. - to generate a color space denoted as ch for channel ch. The overall or final predicted component HDR codeword.

[0120] In some operating scenarios, the 3D-LUT generated from the TPB model may have dark boost issues that increase the brightness level in the predicted HDR codeword values. To address these issues, a 3D-LUT post-processing operation may be performed to manually and / or automatically adjust the 3D-LUT to make dark nodes (or predicted HDR codeword values ​​in dark areas) darker than without the adjustment.

[0121] First, the predicted component RGB HDR codeword values ​​for the same node index (a, b, c) predicted by the static 3D-LUT for all dimensions or channels of the output HDR color space can be collected together as And mapped to component YCbCr HDR codewords via color space conversion RGBtoYCbCr() As shown below:

[0122]

[0123] Therefore, a 3D-LUT for the YCbCr HDR color space can be constructed from the 3D-LUT generated for the output RGB HDR color space.

[0124] Secondly, the dark range along the Y axis / dimension / channel of the YCbCr HDR color space (e.g., the Y codeword value range [0, Y D ) is divided into L intervals. Here, in the overall normalized Y codeword value range (or domain) [0 1], Y D An example value of may be, but is not necessarily limited to, 0.3. The i-th interval (where i = 0, 1, ..., L-1) is in the Y codeword value subrange An example value of L may be, but is not necessarily limited to, 6. For each of the L intervals, all YCbCr entries or nodes having Y (or grayscale) codeword values ​​within the i-th interval may be collected into a matrix denoted as Ω j 's set (generating L such sets), as shown below:

[0125]

[0126] Third, for each set Ω i , static RGB 3D-LUT with collection Ω i The entry or node lookup value corresponding to all YCbCr entries of the nodes in is reduced (the darker the overall RGB node or codeword value corresponds to, the more the component RGB codeword value in each channel or component ch of the overall RGB node or codeword is reduced), as shown below:

[0127] For all {a,b,c}∈Ω i For ch=R,G,B (18)

[0128] An example value of D may be, but is not necessarily limited to, 0.065. The final 3D-LUT including these codeword adjustments Can be used to convert SDR images (such as from Figure 1 The method may further comprise predicting, mapping or converting SDR codeword values ​​in a mapped / converted HDR image corresponding to the enhanced SDR image (the first stage or processing block of the enhanced SDR image) to HDR codeword values ​​mapped / converted in a mapped / converted HDR image corresponding to the enhanced SDR image.

[0129] Figure 3GThe following diagram shows an example HDR image converted by a static 3D-LUT with and without post-processing of darkness adjustment. The resulting HDR image is the same as the one on the left using the 3D-LUT without dark enhancement. The resulting HDR image contains more details in the dark areas than in the dark areas.

[0130] HDR Enhancement

[0131] In order to provide a pleasing appearance in the output picture / image, the camera ISP in the image acquisition device as described herein can be used to perform a variety of different enhancement processing operations. The enhancement processing operations performed using such a camera ISP can include but are not necessarily limited to local contrast and image detail enhancement.

[0132] To achieve or simulate in a mapped or converted HDR image the same or similar visually pleasing appearance as described herein, it is possible to Figure 1 The enhancement module is implemented in a third component or processing block of the HDR image processing unit to produce or output a final HDR image with a relatively visually pleasing appearance. The enhancement module may operate with or without other available solutions to enhance the image. These other available solutions may include unsharp masking, local Laplacian filtering, proprietary or standard image enhancement operations, etc. In some operating scenarios, the techniques described herein may be used to implement a relatively simple and effective solution to enhance an image within the same dynamic range compared to these other available solutions - for example, to increase the resolution of the image from HDR to HDR. Figure 1 The second component or processing block generates a mapped or converted HDR image that is enhanced to a final enhanced mapped or converted HDR image within the same dynamic range or HDR.

[0133] Figure 2C Picture shows Figure 1 An example of the third major component or processing block used for image enhancement in . Figure 2C As shown in , the framework includes, for example, Y (or brightness contrast) local shaping and RGB (or R / G / B saturation) local shaping. Figure 1 Before performing these local shaping operations, a family of hybrid shifted sigmoid functions with linear mapping (HSSL) LUTs can be built or constructed.

[0134] In some operating scenarios, each dimension or color channel in some or all of the dimensions or color channels in which local shaping is to be performed (e.g., any of R / G / B, any of Y / R / G / B, etc.) can use its own unique HSSL LUT that is different from other HSSL LUTs used by other dimensions or color channels.

[0135] In some operating scenarios, each dimension or color channel of some or all dimensions or color channels in which local shaping is to be performed (e.g., any of R / G / B, any of Y / R / G / B, etc.) can use or share the same HSSL LUT used by other dimensions or color channels.

[0136] like Figure 2C As shown in , the input HDR image (denoted as "HDR RGB") can first be converted into an HDR image in the YCbCr domain or color space by a RGB to YCbCr color space transform. The Y or grayscale codewords (e.g., local average Y values ​​or intermediate L values, etc.) of the HDR image in the Y channel can be processed and used to generate a local shaping function (LRF) index map - for example, it can include (e.g., per pixel, per pixel block, etc.) local average Y values ​​or intermediate L values ​​as (e.g., per pixel, per pixel block, etc.) LRF indices.

[0137] like Figure 2C The grayscale shaping module shown in (denoted as "Y Local Shaping") receives or obtains the Y channel codeword value for each pixel in the HDR image in the YCbCr color space, uses the LRF index corresponding to the pixel in the LRF index map to find or select a LUT applicable to the pixel in a family of Y channel HSSL LUTs, and uses the selected LUT to map the Y channel codeword value to an enhanced or locally shaped Y channel codeword value. The enhanced Y channel codeword value can be combined with the (original or non-enhanced or non-locally shaped) Cb and Cr codeword values ​​in the HDR image in the YCbCr color space in a YCbCr to RGB color space transformation of the RGB codeword value of the pixel in the brightness enhanced HDR image returned to the RGB domain or color space.

[0138] The Y channel enhancement operation performed using the Y channel local shaping discussed above may be followed by an R / G / B enhancement operation performed using R / G / B local shaping.

[0139] For each color channel in the RGB color space, such as Figure 2CThe R / G / B shaping module shown in (denoted as "RGB Local Shaping") receives or obtains a per-channel (R or G or B) brightness enhanced codeword value for each pixel of each channel in an HDR image in an RGB color space, uses a per-channel LRF index in an LRF index map corresponding to the pixel and the channel to find or select an applicable LUT for the pixel among a family of per-channel HSSL LUTs for the channel (denoted as RGB HSSL LUTs for simplicity), and uses the selected LUT to map the per-channel brightness enhanced codeword value to the final enhanced or locally shaped per-channel codeword value of the channel in an enhanced or output HDR image (denoted as "Enhanced HDR RGB").

[0140] For illustrative purposes only, it has been described that RGB local shaping may follow Y channel local shaping to generate an enhanced or output HDR image. It should be noted that in other embodiments, Y channel local shaping may follow RGB local shaping to generate an enhanced or output HDR image. Furthermore, optionally or alternatively, in some operating scenarios, only Y channel local shaping may be performed (without separate RGB local shaping) to generate the final enhanced or output HDR image. Furthermore, optionally or alternatively, in some operating scenarios, only RGB local shaping may be performed (without separate Y channel local shaping) to generate the final enhanced or output HDR image.

[0141] Hybrid Shifted S-Shaped (HSSL) LUT with Linear

[0142] A family of HSSL LUTs may be constructed for a single color channel (denoted as ch) or for some or all color channels. This family of HSSL LUTs may include a total of K (e.g., 1024, 4096, etc.) different (HSSL) LUTs or local shaping functions. These different (HSSL) LUTs or local shaping functions may be indexed by LRF index k (from 0 to K-1), respectively. Each LUT may be constructed by merging a shifted sigmoid function and a linear function and characterized by a parameter set as shown in Table 1 below.

[0143] Table 1

[0144]

[0145] Figure 3H An example curve for generating a LUT in a family of HSSL LUTs is shown. As shown in the figure, the curve for generating the LUT may include a left (L) sigmoid function segment and a right (R) sigmoid function segment, which are determined by the parameter c (ch,k) The left and right sigmoid function fragments can be used with parameters and The two sigmoid functions (which may be referred to as left sigmoid function and right sigmoid function without loss of generality) of the two slopes indicated are respectively specified. The left and right sigmoid function segments are further specified relative to the parameters c. (ch,k) The given centers are located at distances and The lower point at and the upper point at are joined with two linear segments (which may be referred to as the left segment and the right segment without loss of generality) having a slope of one (1).

[0146] A sigmoid function used to represent a sigmoid function fragment as described herein may be expressed as a function whose slope is denoted as m, as follows:

[0147]

[0148] For different values ​​of the slope of m, the maximum and minimum values ​​supported by the sigmoid function on x in the value range [0,1] are different.

[0149] Used to specify Figure 3H The maximum and minimum values ​​of the left and right sigmoid function of the left and right sigmoid function fragments can be expressed as follows:

[0150]

[0151] The maximum and minimum values ​​in expression (20) above can be used as normalization factors to define the following left and right scaled sigmoid functions:

[0152] for

[0153] for

[0154] The left and right scaled sigmoid functions in expression (21) can be expressed relative to the center c (ch,k) Shifted to produce left and right shifted sigmoid functions as follows:

[0155] for

[0156] for

[0157] The kth local shaping function (which can be used to generate the corresponding HSSL LUT) can initially be set to a default 1:1 mapping: LRF(ch,k) (x) = x. In addition, the center c of the kth local shaping function (ch,k) Can be set to

[0158] Centerc (ch,k) The default function around LRF (ch,k) The left and right regions of (x) can be replaced by scaled sigmoid functions with left and right shifts in the above expression (22). More specifically, for the left region or segment c (ch,k) )'s default LRF (ch,k) (x) can be used in expression (22-1) instead. Since the left area or segment may be larger than short or small, so it can be calculated as follows The maximum and minimum values ​​of :

[0159]

[0160] if These maxima and minima can then be used in normalization or scaling to avoid generating negative (output) local shaping function values, as shown below:

[0161]

[0162] The final (output) local shaping function value in the left region or segment can be given as follows:

[0163]

[0164] Similarly, for the right region or segment Default local shaping function LRF (ch,k) (x) can be obtained from the expression (22-2) Instead. Since the right region or segment may be shorter or smaller than therefore The maximum and minimum values ​​of can be calculated as follows:

[0165]

[0166] if These maxima and minima can then be used in normalization or scaling to avoid generating negative (output) local shaping function values, as shown below:

[0167]

[0168] The final (output) local shaping function value in the right region or segment can be given as:

[0169]

[0170] Fig. 3I Example local shaping functions (belonging to a family of K = 4096 different local shaping functions) are illustrated, where k = 512, 2048 and 3584, respectively, where As shown in the figure, these local shaping functions can be individually adjusted with the slope of the left sigmoid function segment varies with the slope of the right sigmoid function segment alone or the slopes of the left and right sigmoid function segments and The slope value varies with both. and The larger the value, the stronger the local contrast provided by these local shaping functions.

[0171] Figure 3J Example local shaping functions (belonging to a family of K = 4096 different local shaping functions) are illustrated, where k = 512, 2048 and 3584, respectively, where As shown in the figure, these local shaping functions can only vary with the left region or segment size (such as from the center c (ch,k) Measured) varies only with the size of the right area or segment (e.g., from the center c (ch,k) Measured) or with the left and right area or segment size and Both vary. and The larger it is, the larger the modification area will be.

[0172] Local Reshaping Function (LRF) Index Map

[0173] Once a family of HSSL LUTs or local shaping functions LRFs are generated (ch,k) (x), a local shaping function (LRF) index map can be generated from the input HDR image, and the index map can be used to select different HSSL LUTs or different local shaping functions to shape different pixels in the input HDR image into the final enhanced or locally shaped HDR image.

[0174] By way of illustration but not limitation, taking Y channel local shaping as an example, for each pixel in the input HDR image, the grayscale or brightness of the pixel can be locally shaped or enhanced using an applicable HSSL LUT or local shaping function specially selected for the pixel, thereby increasing the saturation or contrast (brightness difference) among the pixels in the local neighborhood around each pixel.

[0175] As described above, the curves that produce each HSSL LUT or local shaping function in a family of HSSL LUTs or local shaping functions are more or less linear, except for the regions / segments that are replaced with shifted / scaled S-shaped curves. The center of the joining left and right S-shaped curve segments can be located at different positions for each such curve, for example depending on a parameter k corresponding to the LRF index (such as the local mean (grayscale value) or the intermediate L value). Therefore, a local mean can be calculated for a local neighborhood of each pixel and used to select a specific local shaping function, where its center is given as the local mean. Pixels or codeword values ​​within the local neighborhood can use a specific local shaping function to enhance or enhance brightness differences or contrast or saturation levels or differences, thereby enhancing the visual appearance of the final enhanced image.

[0176] The challenge to be solved in selecting the local shaping function using the local average is how to calculate the local average while avoiding generating visual artifacts such as halos that separate different image details of different average brightness levels near edges. As mentioned above, the guided filter is an edge-aware (or edge-preserving) filter and can therefore be applied to select the local average for calculating the local shaping function.

[0177] Denote the i-th pixel of the input HDR image (of size WxH, where W is a non-zero positive integer representing the total number of pixels in a row and H is a non-zero positive integer representing the total number of pixels in a column) as v i .

[0178] Various different types of filters used to generate the LRF index map include Gaussian filters, which are represented as follows:

[0179] Y=GAUSS(X,σ g ) (29)

[0180] Where Y represents the pre-filtered image / graph X with standard deviation σ g The Gaussian blurred or filtered image / graph obtained after Gaussian filtering of .

[0181] In some operation scenarios, Gaussian blur or filtering can be accelerated, for example, using an iterative box filter to approximate a Gaussian filter. An example of using an iterative box filter to approximate a Gaussian filter is described in "A Survey of Gaussian Convolution Algorithms" by Pascal Getreuer, Image Processing On Line, Vol. 3, pp. 286-310 (2013); "Efficient Synthesis of Gaussian Filters by Cascaded Uniform Filters" by William M. Wells, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 8, No. 2, pp. 234-239 (1986), the contents of which are incorporated herein by reference in their entirety.

[0182] The gradient filter can be used to calculate the local gradient Y of an image or graph X as follows:

[0183] Y=GRAD(X) (30)

[0184] In some operating scenarios, in order to generate local gradients in an image or graph, a Sobel filter can be used as a gradient filter. Example local reshaping operations such as those associated with Gaussian blur, Sobel, and multi-stage guided filters are described in U.S. Provisional Patent Application No. 63 / 086,699, filed by Tsung-Wei Huang et al. on October 2, 2020, entitled “ADAPTIVE LOCAL RESHAPING FOR SDR-TO-HDR UP-CONVERSION,” the contents of which are incorporated herein by reference in their entirety.

[0185] The guided filter can be used to generate a guided filtered image or map Y of the image or map X as follows:

[0186] Y=GUIDE(X,I,B g ) (31)

[0187] Where I represents the guide image, and B g Represents the dimension of the local neighborhood used in guided filtering.

[0188] In some operation scenarios, multi-stage (T stages in total) filtering may be performed to generate an LRF index map.

[0189] A common feature map (or image) can be first created for all stages in the multi-stage filtering as follows:

[0190]

[0191] This common feature map (or image) can be used by each stage of the multi-stage filtering to generate a corresponding filtered map according to the example process shown in Table 2 below.

[0192] Table 2

[0193]

[0194] All these multiple levels of filtered images The weighted combination of can be generated as follows:

[0195]

[0196] where w (t) Represents the weighting factor used to obtain the normalized LRF index within the normalized LRF value range [0,1].

[0197] The LRF map index of the t-th pixel can be calculated as follows:

[0198]

[0199] where f i ∈[0,1,…,K-1].

[0200] As a non-limiting example, T = 2.w (1) =0.7, w (2) =0.3, Local shaping mechanism

[0201] Given a family of HSSL LUTs or local shaping functions {LRF (cj,k) (x)} and the LRF index map generated from the input HDR image and the LRF index map {f i}, local shaping may be performed on an input HDR codeword of an input HDR image to generate corresponding locally shaped HDR codewords for a color channel. The locally shaped HDR codeword for color channel ch may be combined with or further enhanced with other locally shaped HDR codewords in one or more other color channels to generate a final or enhanced HDR codeword in some or all color channels of a final or enhanced HDR image.

[0202] More specifically, for a color channel ch, given the input HDR pixel or codeword value v for the i-th pixel i , we can use the LRF index map of the i-th pixel {f i The LRF index f in} iFrom a family of HSSL LUTs or local shaping functions {LRF (ch,k) (x)} to select or find the HSSL LUT or is represented as The local shaping function of . It can be expressed as The local reshaped HDR pixel or codeword can use the input HDR pixel or codeword value v of the i-th pixel i Use the selected HSSL LUT or local shaping function Generates as follows:

[0203]

[0204] As described above, the locally shaped HDR pixel or codeword value may be further locally shaped or enhanced in (one or more) other color channels. For each LRF index k supported in a family of HSSL LUTs or local shaping functions as described herein, the operating parameters for controlling the amount of local shaping enhancement (e.g., slope, region / segment size, etc.) for a given channel ch may include, but are not limited to, some or all of the following: {g R (ch,k)},{m L (ch,k)}; {m R (ch,k)}.

[0205] In some operational scenarios, a computing device (such as an image acquisition device, mobile device, camera, etc.) that performs image conversion and / or enhancement operations as described herein may be configured or preconfigured with operational or functional parameters for each of some or all of the local shaping functions in a specified family.

[0206] Furthermore, optionally or alternatively, in some operational scenarios, each of some or all of the local shaping functions in a family may be represented by a corresponding (HSSL) LUT generated by a corresponding local shaping function in a family of local shaping functions, thereby producing a corresponding family of HSSL LUTs generated from the corresponding family of local shaping functions. A computing device (such as an image acquisition device, mobile device, camera, etc.) that performs image conversion and / or enhancement operations as described herein may be configured or preconfigured with operational or functional parameters that specify each of some or all of the HSSL LUTs in the family of HSSL LUTs generated from the corresponding family of local shaping functions.

[0207] Example Processing Flow

[0208] Figure 4BAn example processing flow according to an embodiment is illustrated. In some embodiments, one or more computing devices or components (e.g., encoding device / module, transcoding device / module, decoding device / module, inverse tone mapping device / module, tone mapping device / module, media device / module, inverse mapping generation and application system, etc.) can perform this processing flow. In block 452, the image processing system applies guided filtering to a first image of a first dynamic range using a camera raw image as a guide image to generate an intermediate image of a first dynamic range, wherein the first image of the first dynamic range is generated from the camera raw image using an image signal processor of an image acquisition device.

[0209] In block 454 , the image processing system performs dynamic range mapping on the intermediate image of the first dynamic range to generate a second image of a second dynamic range different from the first dynamic range.

[0210] In block 456 , the image processing system uses the second image of the second dynamic range to generate a specific local shaping function index value for selecting a specific local shaping function for the second image of the second dynamic range.

[0211] In block 458 , the image processing system applies the particular local shaping function to the second image of the second dynamic range to generate a locally shaped image of the second dynamic range.

[0212] In one embodiment, the specific local shaping function index values ​​collectively form a local shaping function index value map.

[0213] In one embodiment, the first dynamic range represents a standard dynamic range; and the second dynamic range represents a high dynamic range.

[0214] In one embodiment, each local shaping function index value in a particular local shaping function index value is used to look up a corresponding local shaping function in a family of local shaping functions.

[0215] In one embodiment, each of the specific local shaping functions is represented by a corresponding lookup table.

[0216] In an embodiment, each of the particular local shaping functions represents a hybrid shifted sigmoid function.

[0217] In an embodiment, each of the specific local shaping functions shapes component codewords in a color channel derived from the second image into locally shaped component codewords in the color channel.

[0218] In one embodiment, the guided filtering includes a first guided filtering having a relatively small spatial kernel and a second guided filtering having a relatively large spatial kernel.

[0219] In one embodiment, a first guided filter with a relatively small spatial kernel is used to recover image details associated with edges that are in the camera raw image but lost in the first image; a second guided filter with a relatively large spatial kernel is used to recover image details associated with dark and bright areas that are in the camera raw image but lost in the first image.

[0220] In one embodiment, the intermediate image is fused by a first guided filtered image generated from a first guided filtering having a relatively small spatial kernel and a second guided filtered image generated from a second guided filtering having a relatively large spatial kernel.

[0221] In an embodiment, a smoothed binarized local standard deviation map is generated from the first image; and weighting factors for fusing the first guided filtered image and the second guided filtered image are determined based at least in part on the smoothed binarized local standard deviation map.

[0222] In one embodiment, the dynamic range map represents a tensor product B-spline (TPB) map.

[0223] In one embodiment, the dynamic range mapping represents a static mapping specified using optimized operating parameters generated during a model training phase, wherein the optimized operating parameters are generated using training images of the first dynamic range and the second dynamic range obtained by spatially aligning captured images of the first dynamic range and the second dynamic range.

[0224] In one embodiment, a computing device (such as a display device, a mobile device, a set-top box, a multimedia device, etc.) is configured to perform any of the above methods. In one embodiment, an apparatus includes a processor and is configured to perform any of the above methods. In one embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause any of the above methods to be performed.

[0225] In one embodiment, a computing device includes one or more processors and one or more storage media storing a set of instructions that, when executed by the one or more processors, cause performance of any of the methods described above.

[0226] Note that, although separate embodiments are discussed herein, any combination of embodiments and / or portions of embodiments discussed herein may be combined to form further embodiments.

[0227] Example Computer System Implementation

[0228] Embodiments of the present invention may be implemented using a computer system, a system configured in an electronic circuit system and components, an integrated circuit (IC) device (such as a microcontroller, a field programmable gate array (FPGA) or another configurable or programmable logic device (PLD)), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or a device including one or more such systems, devices or components. The computer and / or IC may execute, control or perform instructions related to adaptive perceptual quantization of images with enhanced dynamic range, such as the instructions described herein. The computer and / or IC may calculate any of the various parameters or values ​​related to the adaptive perceptual quantization process described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.

[0229] Certain implementations of the present invention include a computer processor that executes software instructions that cause the processor to perform the method of the present disclosure. For example, one or more processors in a display, an encoder, a set-top box, a transcoder, etc. can implement the method related to adaptive perceptual quantization of HDR images as described above by executing software instructions in a program memory accessible to the processor. Embodiments of the present invention can also be provided in the form of a program product. The program product may include any non-transient medium that carries a set of computer-readable signals including instructions, which, when executed by a data processor, cause the data processor to perform the method of an embodiment of the present invention. The program product according to an embodiment of the present invention may take any of a variety of forms. The program product may include, for example, a physical medium, such as a magnetic data storage medium, including a floppy disk, a hard disk drive, an optical data storage medium (including a CD ROM, a DVD), an electronic data storage medium (including a ROM, a flash RAM, etc.). The computer-readable signal on the program product may be optionally compressed or encrypted.

[0230] When a component (e.g., a software module, a processor, a component, a device, a circuit, etc.) is mentioned above, unless otherwise indicated, reference to the component (including reference to "means") should be interpreted as including any component that performs the function of the component as an equivalent of the component (e.g., functionally equivalent), including components that are not structurally equivalent to the structure for performing the function disclosed in the illustrated example embodiments of the present invention.

[0231] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. The special-purpose computing device can be hard-wired to perform the technology, or can include a digital electronic device permanently programmed to perform the technology, such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs), or can include one or more general-purpose hardware processors programmed to perform the technology according to program instructions in firmware, memory, other storage devices, or combinations. Such special-purpose computing devices can also combine customized hard-wired logic, ASICs or FPGAs with customized programming to implement the technology. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that combines hard-wired and / or program logic to implement the technology.

[0232] For example, Figure 5 is a block diagram illustrating a computer system 500 upon which embodiments of the present invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled with bus 502 for processing information. Hardware processor 504 may be, for example, a general purpose microprocessor.

[0233] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 504. When such instructions are stored in a non-transitory storage medium accessible to processor 504, they turn computer system 500 into a special-purpose machine customized to perform the operations specified in the instructions.

[0234] Computer system 500 also includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504. A storage device 510, such as a magnetic or optical disk, is provided and coupled to bus 502 for storing information and instructions.

[0235] The computer system 500 may be coupled to a display 512, such as a liquid crystal display, via the bus 502 for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to the bus 502 for communicating information and command selections to the processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating direction information and command selections to the processor 504 and for controlling cursor movement on the display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to specify a position in a plane.

[0236] Computer system 500 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that, in combination with the computer system, makes computer system 500 a special purpose machine or programs the computer system to be a special purpose machine. According to one embodiment, the techniques described herein are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Executing the sequences of instructions contained in main memory 506 causes processor 504 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0237] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or cassette tapes.

[0238] Storage media are distinct from, but may be used in conjunction with, transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that make up bus 502. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0239] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 may receive the data on the telephone line and use an infrared transmitter to convert the data into an infrared signal. An infrared detector may receive the data carried in the infrared signal and appropriate circuitry may place the data on the bus 502. The bus 502 carries the data to the main memory 506, from which the processor 504 retrieves and executes the instructions. The instructions received by the main memory 506 may optionally be stored on the storage device 510 before or after execution by the processor 504.

[0240] The computer system 500 also includes a communication interface 518 coupled to the bus 502. The communication interface 518 provides a two-way data communication coupled to a network link 520 connected to a local network 522. For example, the communication interface 518 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 518 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. A wireless link can also be implemented. In any such implementation, the communication interface 518 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0241] The network link 520 typically provides data communication to other data devices through one or more networks. For example, the network link 520 can provide a connection to a host computer 524 or to data equipment operated by an Internet Service Provider (ISP) 526 through a local network 522. The ISP 526 in turn provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 528. Both the local network 522 and the Internet 528 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 520 and through the communication interface 518, which carry the digital data to and from the computer system 500, are example forms of transmission media.

[0242] Computer system 500 can send messages and receive data (including program code) through the network(s), network link 520, and communication interface 518. In the Internet example, server 530 can transmit the requested code for an application through Internet 528, ISP 526, local network 522, and communication interface 518.

[0243] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510 or other non-volatile storage for later execution.

[0244] Equivalents, Extensions, Alternatives and Miscellaneous

[0245] In the foregoing description, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Therefore, the only and exclusive indication of what is claimed as an embodiment of the present invention and what the applicant wishes to be the claimed embodiment of the present invention is the set of claims issued from this application in the specific form in which these claims are issued, including any subsequent corrections. Any definitions of terms contained in such claims expressly set forth herein shall govern the meaning of such terms used in the claims. Therefore, any limitation, element, characteristic, feature, advantage or attribute not expressly stated in a claim should not limit the scope of such claim in any way. Therefore, the description and drawings should be regarded as illustrative and not restrictive.

[0246] Example embodiments listed

[0247] The present invention may be implemented in any form described herein, including but not limited to the following enumerated example embodiments (EEEs), which describe the structure, features, and functionality of some portions of embodiments of the present invention.

[0248] EEE1. A method comprising:

[0249] applying a guided filter to a first image of a first dynamic range using a camera raw image as a guided image to generate an intermediate image of a first dynamic range, the first image of the first dynamic range being generated from the camera raw image using an image signal processor of an image acquisition device;

[0250] performing dynamic range mapping on the intermediate image of the first dynamic range to generate a second image of a second dynamic range different from the first dynamic range;

[0251] generating a specific local shaping function index value for selecting a specific local shaping function for the second image of the second dynamic range using the second image of the second dynamic range;

[0252] The specific local shaping function is applied to the second image of the second dynamic range to generate a locally shaped image of the second dynamic range.

[0253] EEE2. A method as described in EEE1, wherein specific local shaping function index values ​​collectively form a local shaping function index value map.

[0254] EEE3. A method as described in EEE1 or EEE2, wherein the first dynamic range represents a standard dynamic range; wherein the second dynamic range represents a high dynamic range.

[0255] EEE4. A method as described in any one of EEE1-EEE3, wherein each local shaping function index value in a specific local shaping function index value is used to find a corresponding local shaping function in a family of local shaping functions.

[0256] EEE5. A method as described in any one of EEE1-EEE4, wherein each local shaping function in the specific local shaping functions is represented by a corresponding lookup table.

[0257] EEE6. A method as described in any one of EEE1-EEE5, wherein each of the specific local shaping functions represents a mixed shifted S-shaped function.

[0258] EEE7. A method as described in EEE6, wherein the mixed shifted S-shaped function is constructed by joining at the midpoint a first S-shaped function having a first segment length and a first slope located to the left of the midpoint and a second S-shaped function having a second segment length and a second slope located to the right of the midpoint.

[0259] EEE8. A method as described in EEE7, wherein each of the specific local shaping functions is selected from a family of local shaping functions; wherein each local shaping function in this family of local shaping functions is indexed using a corresponding local shaping function index value within an index value range; wherein each local shaping function in this family of local shaping functions is generated based on a mixed shifted S-shaped function based on different combinations of values ​​of a midpoint, a first segment length, a first slope, a second segment length and a second slope.

[0260] EEE9. A method as described in any one of EEE1-EEE8, wherein each of the specific local shaping functions shapes the component codewords in the color channel obtained from the second image into locally shaped component codewords in the color channel.

[0261] EEE10. A method as described in any one of EEE1-EEE9, wherein the guided filtering includes a first guided filtering having a relatively small spatial kernel and a second guided filtering having a relatively large spatial kernel. EEE10.

[0262] EEE11. A method as described in EEE10, wherein a first guided filter having a relatively small spatial kernel is configured to recover image details related to edges that are in the camera's raw image but lost in the first image; wherein a second guided filter having a relatively large spatial kernel is configured to recover image details related to dark areas and bright areas that are in the camera's raw image but lost in the first image.

[0263] EEE12. A method as described in EEE10 or EEE11, wherein the intermediate image is fused by a first guided filtered image generated from a first guided filtering having a relatively small spatial kernel and a second guided filtered image generated from a second guided filtering having a relatively large spatial kernel.

[0264] EEE13. A method as claimed in claim 12, wherein a smoothed binarized local standard deviation map is generated from the first image; wherein a weighting factor for fusing the first guided filtered image and the second guided filtered image is determined at least in part based on the smoothed binarized local standard deviation map.

[0265] EEE14. A method as described in any one of EEE1-EEE13, wherein the dynamic range mapping represents a tensor product B-spline (TPB) mapping.

[0266] EEE15. A method as described in any one of EEE1-EEE14, wherein the dynamic range mapping represents a static mapping specified using optimized operating parameters generated in a model training phase, wherein the optimized operating parameters are generated using training images of the first dynamic range and the second dynamic range obtained by spatially aligning captured images of the first dynamic range and the second dynamic range.

[0267] EEE16. A method as described in any one of EEE1-EEE15, wherein the dynamic range mapping is represented by a three-dimensional lookup table (3D-LUT), the three-dimensional lookup table comprising a plurality of entries mapping an input three-dimensional codeword to a predicted three-dimensional codeword in a mapped RGB color space; the method further comprising:

[0268] converting predicted RGB codewords in the plurality of entries in the 3D-LUT into corresponding YCbCr codewords in a YCbCr color space;

[0269] Divide the corresponding YCbCr codeword into multiple Y-dimension partitions along the Y-dimension of the YCbCr color space;

[0270] identifying, among the plurality of entries in the 3D-LUT, a subset of entries having dark brightness levels based on the plurality of Y-dimension partitions of the corresponding YCbCr codewords;

[0271] The predicted three-dimensional codewords in the subset of entries are adjusted according to the dimming level.

[0272] EEE17. A device comprising a processor and configured to perform any method as described in EEE1-EEE16.

[0273] EEE18. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, the computer-executable instructions being used to use one or more processors to execute a method according to any one of the methods described in EEE1-EEE16.

[0274] EEE19. A computer system configured to perform any method as described in EEE1-EEE16.

Claims

1. A method comprising: applying a guided filter to a first image of a first dynamic range using a camera raw image as a guided image to generate an intermediate image of a first dynamic range, the first image of the first dynamic range being generated from the camera raw image using an image signal processor of an image acquisition device; performing dynamic range mapping on the intermediate image of the first dynamic range to generate a second image of a second dynamic range different from the first dynamic range; generating a specific local shaping function index value for selecting a specific local shaping function for the second image of the second dynamic range using the second image of the second dynamic range; The specific local shaping function is applied to the second image of the second dynamic range to generate a locally shaped image of the second dynamic range.

2. The method of claim 1, wherein: The specific local shaping function index values ​​collectively form a local shaping function index value map.

3. The method according to claim 1 or 2, wherein: The first dynamic range represents a standard dynamic range; wherein the second dynamic range represents a high dynamic range.

4. The method according to any one of claims 1 to 3, wherein: Each of the specific local shaping functions represents a hybrid shifted sigmoid function.

5. The method of claim 4, wherein: The hybrid shifted sigmoid function is constructed by joining at the midpoint a first sigmoid function having a first segment length and a first slope to the left of the midpoint and a second sigmoid function having a second segment length and a second slope to the right of the midpoint.

6. The method of claim 5, wherein: Each of the specific local shaping functions is selected from a family of local shaping functions; wherein each of the local shaping functions in this family of local shaping functions is indexed using a corresponding local shaping function index value within an index value range; wherein each of the local shaping functions in this family of local shaping functions is generated based on a mixed shifted S-shaped function based on different combinations of values ​​of a midpoint, a first segment length, a first slope, a second segment length and a second slope.

7. The method according to any one of claims 1 to 6, wherein: The guided filtering includes a first guided filtering having a relatively small spatial kernel and a second guided filtering having a relatively large spatial kernel.

8. The method of claim 7, wherein: A first guided filter having a relatively small spatial kernel is configured to recover image details associated with edges that are in the camera raw image but lost in the first image; wherein a second guided filter having a relatively large spatial kernel is configured to recover image details associated with dark and bright areas that are in the camera raw image but lost in the first image.

9. The method according to claim 7 or 8, wherein: The intermediate image is fused from a first guided filtered image generated from a first guided filtering having a relatively small spatial kernel and a second guided filtered image generated from a second guided filtering having a relatively large spatial kernel.

10. The method of claim 9, wherein: A smoothed, binarized local standard deviation map is generated from the first image; wherein weighting factors for fusing the first guided filtered image and the second guided filtered image are determined based at least in part on the smoothed, binarized local standard deviation map.

11. The method according to any one of claims 1 to 10, wherein: The dynamic range map represents a tensor product B-spline (TPB) map.

12. The method according to any one of claims 1 to 11, wherein: The dynamic range mapping represents a static mapping specified using optimized operating parameters generated during a model training phase, wherein the optimized operating parameters are generated using training images of the first dynamic range and the second dynamic range obtained by spatially aligning captured images of the first dynamic range and the second dynamic range.

13. The method according to any one of claims 1 to 12, wherein: The dynamic range mapping is represented by a three-dimensional lookup table 3D-LUT, the 3D-LUT comprising a plurality of entries mapping an input three-dimensional codeword to a predicted three-dimensional codeword in a mapped RGB color space; the method further comprising: converting predicted RGB codewords in the plurality of entries in the 3D-LUT into corresponding YCbCr codewords in a YCbCr color space; Divide the corresponding YCbCr codeword into multiple Y-dimension partitions along the Y-dimension of the YCbCr color space; identifying, among the plurality of entries in the 3D-LUT, a subset of entries having dark brightness levels based on the plurality of Y-dimension partitions of the corresponding YCbCr codewords; The predicted three-dimensional codewords in the subset of entries are adjusted according to the dimming level.

14. An apparatus comprising a processor and configured to perform the method according to any one of claims 1-13.

15. A non-transitory computer-readable storage medium having computer-executable instructions stored thereon, which, when executed on one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 13.

Images