RAW image data reconstruction system and method

Abstract

A method for reconstructing a RAW image includes generating a low-frequency image and a high-frequency image from an initial image, linearly estimating the high-frequency image to generate a reconstructed high-frequency image, sparsely interpolating the low-frequency image to generate a reconstructed low-frequency image, and generating a reconstructed RAW image from the reconstructed low-frequency image and the reconstructed high-frequency image.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of priority to European Patent Application No. 22166487.3 (Reference No. D21137EP) and U.S. Provisional Patent Application No. 63 / 326,987 (Reference No. D21137USP1), both filed on April 4, 2022, the entire text of each of which is incorporated by reference. [Background technology]

[0002] Many consumer electronics products include at least one camera. These products include tablet computers, mobile phones, and smart watches. In such products, as well as digital still cameras themselves, the camera includes an image sensor having many pixels arranged as a pixel array. An image sensor, an image signal processor (ISP), or an image editing tool can adjust the captured RAW image to more accurately depict what the human eye sees. Many of the operations performed by the ISP or editing tool are nonlinear and spatially varying, requiring intensive computational power to be reversed, if at all. As recognized by the present inventors, improved techniques for RAW image reconstruction are desirable.

[0003] Mahmoud Afifi's "Image color correction, enhancement, and editing" (Arxiv.org, Cornell University Library, NY 14853, 28 July 2021, XP091018402) discloses methods and approaches for image color correction, color enhancement, and color editing. The color correction problem is studied from the perspective of a camera's image signal processor (ISP). The camera's ISP is hardware that applies a series of in-camera image processing and color manipulation steps, many of which are nonlinear in nature, to render the initial sensor image into a final, finished photographic representation stored in the 8-bit standard RGB (sRGB) color space. Because white balance (WB) is one of the primary processes applied by the ISP for color correction, two different methods of ISP white balancing are presented. Other scenarios for correcting and editing image color are also discussed. A set of methods is presented for correcting and editing the WB settings of images that have been improperly white balanced by the ISP. This is followed by a discussion of other factors that significantly affect the quality of camera-rendered colors. Two different methods are outlined for correcting exposure errors in camera-rendered images. Finally, automatic post-capture color editing and manipulation is discussed. In particular, an automatic image restoration method is proposed that generates different, more realistic versions of the same camera-rendered image using new colors.

[0004] Abhijith Punnappurath et al.'s "Spatially Aware Metadata for Raw Reconstruction" (2021 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 3 January 2021, pages 218-226, XP033926466) discloses a RAW reconstruction method based on spatially aware metadata. After a camera sensor captures a RAW-RGB image, the image is processed into a standard RGB (sRGB) image through a series of on-board operations performed by the camera's image signal processor (ISP). Among these processing steps, local tone mapping is one of the most important operations used to improve the overall appearance of the final rendered sRGB image. For certain applications, it is often desirable to de-render the sRGB image, i.e., return it to its original RAW-RGB values ​​by rendering it unprocessed. This "RAW reconstruction" is a challenging task because many of the operations performed by the ISP, including local tone mapping, are nonlinear and difficult to reverse. Existing RAW reconstruction methods that store special metadata at capture time to enable RAW restoration ignore local tone mapping and assume a global transformation between the RAW-RGB color space and the sRGB color space. Our spatially aware metadata-based RAW reconstruction method is robust to local tone mapping and provides significantly higher RAW reconstruction accuracy (average PSNR improvement of 6 dB) compared to existing RAW reconstruction methods. Our method requires only 0.2% samples of the full-size image as metadata, incurs negligible computational overhead at capture time, and can be easily integrated into modern ISPs.

[0005] Nguyen Rang et al., "RAW Image Reconstruction Using a Self-Contained sRGB-JPEG Image with Only 64 KB Overhead," (2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, 27 June 2016, pages 1655–1663, XP033021343), discloses a method for reconstructing a RAW image from an sRGB-JPEG image. Most camera images are saved as 8-bit standard RGB (sRGB) compressed JPEGs. Even when JPEG compression is set to its highest quality, the encoded sRGB image undergoes significant processing in terms of color and tone manipulation. This makes sRGB-JPEG images undesirable for many computer vision tasks that assume a direct relationship between pixel values ​​and incident light. For such applications, the RAW image format is preferred because it represents a minimally processed, sensor-specific RGB image with a higher dynamic range that is linear with respect to scene luminance. However, a drawback of RAW images is that they require a large amount of storage space and are not well supported in many imaging applications. To address this issue, we present a method for encoding the necessary metadata in sRGB images to reconstruct high-quality RAW images. The approach disclosed herein does not require camera calibration, can reconstruct the original RAW within 0.3% error, and involves only 64 KB of overhead for additional data. More importantly, the output is a fully embedded, 100% compliant sRGB-JPEG file that can be used as is, does not affect any existing image workflow, and the RAW image can be extracted when needed and ignored when not needed. Summary of the Invention

[0006] The invention is defined by the independent claims. The dependent claims relate to optional features in some embodiments. According to one embodiment, a method for reconstructing RAW image data includes generating a low-frequency image and a high-frequency image from an initial image, linearly estimating the high-frequency image to generate a reconstructed high-frequency image, sparsely interpolating the low-frequency image to generate a reconstructed low-frequency image, and generating a reconstructed RAW image from the reconstructed low-frequency image and the reconstructed high-frequency image.

[0007] According to one embodiment, a method for generating an image reconstruction metadata set includes generating a RAW low-frequency image and a RAW high-frequency image from a RAW image or an image derived from the RAW image using a set of decomposition parameters; subsampling the RAW low-frequency image to generate subsampled data; obtaining a high-frequency image by filtering a rendered image using the set of decomposition parameters, the rendered image being derived from the RAW image; and determining a reconstruction matrix whose product with the high-frequency image is equal to the RAW high-frequency image.

[0008] According to an embodiment, a method for reconstructing an image includes generating a RAW low-frequency image and a RAW high-frequency image from the RAW image or an image derived from the RAW image using a set of decomposition parameters; subsampling the RAW low-frequency image to generate sub-sampled data; obtaining a high-frequency image by filtering a rendered image using the set of decomposition parameters, the rendered image being derived from the RAW image; determining a reconstruction matrix whose product with the reconstruction matrix equals the RAW high-frequency image, thereby generating the sub-sampled data and a reconstruction matrix; sparsely interpolating the low-frequency image using the sub-sampled data to generate a reconstructed low-frequency image; linearly estimating the RAW high-frequency image that gives the high-frequency image multiplied by the reconstruction matrix; and generating a reconstructed RAW image from the reconstructed low-frequency image and the reconstructed high-frequency image.

[0009] According to one embodiment, a system includes a processor and a memory communicatively connected to the processor that stores machine-readable instructions that, when executed by the processor, cause the processor to generate a low-frequency image and a high-frequency image from an initial image, linearly estimate the high-frequency image to generate a reconstructed high-frequency image, sparsely interpolate the low-frequency image to generate a reconstructed low-frequency image, and generate a reconstructed RAW image from the reconstructed low-frequency image and the reconstructed high-frequency image. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates a camera imaging a scene with high dynamic range luminance, according to an embodiment.

[0011] [Figure 2] FIG. 2 is a flow diagram illustrating a camera sending the rendered image from FIG. 1 along with generated reconstruction metadata to another electronic device, according to an embodiment.

[0012] [Figure 3] FIG. 10 is a flow diagram illustrating a process for reconstructing a RAW image using a rendered image and generated reconstruction metadata, according to an embodiment.

[0013] [Figure 4] FIG. 2 is a schematic block diagram of a reconstruction metadata generator for generating an image reconstruction metadata set, according to an embodiment.

[0014] [Figure 5] 5 is a schematic diagram of a RAW image data reconstructor that reconstructs a RAW image from a rendered image using the image reconstruction metadata set of FIG. 4 according to an embodiment.

[0015] [Figure 6] 1 is a flowchart illustrating a method for reconstructing a RAW image from an initial image, according to an embodiment.

[0016] [Figure 7] 1 is a flowchart illustrating a method for generating an image reconstruction metadata set, according to an embodiment.

[0017] [Figure 8] 6 is a schematic diagram of a RAW image data reconstructor including the reconstruction metadata generator of FIG. 4 and the RAW image data reconstructor of FIG. 5 according to an embodiment.

[0018] [Figure 9] 1 is a flowchart illustrating a method for reconstructing a RAW image from a rendered image and reconstruction metadata. DETAILED DESCRIPTION OF THE INVENTION

[0019] In modern digital cameras and smart devices (e.g., smartphones), an image sensor captures a real scene and converts it into the camera's RAW-RGB signals. These RAW-RGB signals are also called scene-referred data. An image signal processor (ISP) (hereinafter referred to interchangeably as "processor") processes the RAW-RGB signals (hereinafter referred to interchangeably as "RAW image") to obtain a final output signal. These final output signals are typically called display-referred (hereinafter referred to interchangeably as "rendered image" and "initial image") and may exist in a standard color space such as sRGB, P3, or Bt2020. The final output signal is typically compressed (but may also be uncompressed) and stored in some standard image or video file format such as JPEG, MPEG, AVC, or HEVC.

[0020] To render the RAW-RGB signal into a realistic scene (e.g., mimicking what the human eye can perceive), the ISP typically performs a series of computationally intensive image operations, such as black-level correction, lens shading correction, white balancing, color correction, global / local tone mapping, noise reduction, sharpening, and / or other forms of image or video enhancement. Camera ISPs are typically proprietary and have different sets of parameters that adjust settings established by different camera manufacturers. Therefore, even if one camera captures the same scene with the same field of view, the output image / video may sometimes produce a completely different perception in terms of color, contrast, clarity, noise, detail, etc., from another camera. Some ISP operations are nonlinear and spatially varying, making it difficult to inversely reconstruct the RAW-RGB signal. However, in some applications, it is desirable to reverse the ISP processing and return it to the RAW signal domain. For example, digital cameras are typically adjusted for visually pleasing or aesthetic purposes, sometimes exaggerating saturation or enhancing memory colors. For accurate color reproduction of scene-referred images / data, it is very important to remove extra ISP photo-finishing processes such as saturation enhancement, selective color enhancement, etc. Another example is that sometimes users need to have consistent scene-referred images / data even when real-world scenes are captured from different electronic devices. Once users have the camera's RAW image, they can re-render the RAW image according to their own settings to ensure consistent output.

[0021] Further applications include image manipulations that operate in linear space, such as post-correction of color / white balance. Cameras typically use in-camera auto-white-balancing (AWB) algorithms to estimate scene illumination at the time of capture. However, the AWB algorithms do not always correctly estimate scene illumination and may therefore apply incorrect white balance gains to the RAW image, which can cast undesirable colors in the final output image. While post-correcting these color errors is straightforward in the linear RAW image domain, correcting them in the final output image domain is very difficult because of the nonlinear spatial variable manipulations required for final output.

[0022] Camera RAW image reconstruction may find important applications amid the recent growth and expansion of deep learning training data. This typically requires simulating the same scene under different lighting and different image processing. Most simulations need to operate in linear RAW space to improve results. Even if users can save RAW images along with the final output (e.g., JPEG / HEIC images) at capture time for subsequent processing, the large size of RAW images typically makes storage and communication prohibitive. Therefore, generating information (reconstruction metadata) (e.g., small-sized data) during rendering to reverse the ISP rendering and reconstruction of scene-referred images / data directly from the output image / video is crucial for in-camera image / video processing and offline image / video processing / editing tools, as well as for image / video storage and communication. While the present embodiments discuss JPEG images in sRGB space, these embodiments are used for illustrative purposes. The present embodiments work with any color space and any existing image / video file format. For example, the present embodiments can be used with an iPhone that outputs HEIC images in P3 color space. Additionally, embodiments of the present application can also be extended to R.2020PQ or HLG for HDR capture.

[0023] FIG. 1 shows a camera 130 capturing an image of a scene 120. The scene 120 includes a person 121. The camera 130 includes an imaging lens (not shown), an image sensor 132, a memory 110, and a processor 140 communicatively coupled to the image sensor 132. The memory 110 may be transient and / or non-transitory and may include one or both of volatile memory (e.g., SRAM, DRAM, computational RAM, other volatile memory, or any combination thereof) and non-volatile memory (e.g., FLASH, ROM, magnetic media, optical media, other non-volatile memory, or any combination thereof). Some or all of the memory 110 may be integrated within the processor 140. The processor 140 may be a dedicated image signal processor or a general-purpose processor (e.g., GPU, DSP, microprocessor, etc.). The image sensor 132 includes a pixel array 134A. The pixel array 134A may have a color filter array (CFA) 136 thereon. Pixel array 134A includes a plurality of pixels 134, which are not shown in FIG. 1 for clarity. Each color filter of CFA 136 is aligned with a respective pixel 134 of pixel array 134A. An imaging lens images scene 120 onto image sensor 132. Image sensor 132 also includes circuitry 138 including at least one analog-to-digital converter.

[0024] Each pixel has a respective pixel charge corresponding to a respective intensity of light from the scene 120 imaged on the pixel array 134A. The circuit 138 converts each pixel charge into a respective value of a plurality of pixel values ​​194 (e.g., a RAW image 202, see FIG. 2 ), which are stored in the memory 110. The camera 130 may include a display 139 configured to display the pixel values ​​194 (e.g., a rendered image 204) as an initially captured image 296. The display 139 may display the pixel values ​​194 as a live preview. To generate the rendered image 204, the processor 140 may perform a series of operations on the RAW image in accordance with instructions stored in the memory 110 (described below with reference to FIG. 6 ). The processor 140 also generates reconstruction metadata (e.g., subsampling data, a reconstruction matrix, dynamic range compression (DRC) parameters, and decomposition parameters) used to generate the rendered image 204 and to reconstruct the RAW image from the rendered image, as described below. The ISP 141 may generate the rendered image. The generation of the rendered image and reconstruction metadata, as well as the reconstruction of the RAW image, is described below.

[0025] FIG. 2 is a flow diagram 200 illustrating camera 130 transmitting a RAW image and / or a rendered image and reconstruction metadata to another electronic device. Camera 130 can transmit either the RAW image 202 and / or the rendered image and reconstruction metadata 204 to another electronic device 230 (e.g., a laptop, desktop, etc.). Laptop 230 can use the RAW image 202 to create the rendered image and reconstruction metadata 204. For example, a user of laptop 230 may edit the RAW image 202 to create an aesthetically pleasing rendered image. Alternatively, the user of laptop 230 may wish to modify the rendered image, and because nonlinear processing was performed to create the rendered image, modifying the image may require initiating a new reconstruction of RAW image 202. Furthermore, laptop 230 may transmit the RAW image 202 (or the reconstructed RAW image) or the rendered image and reconstruction metadata 204 to yet another electronic device 232. Similarly, the electronic device 232 may use the RAW image 202 to create a rendered image, use the reconstructed RAW image to create another rendered image, or use the rendered image and reconstruction metadata 204 to reconstruct a RAW image.

[0026] 2 is for illustrative purposes and is not intended to limit the scope of the present application. For example, camera 130 can transmit raw image 202, as well as rendered image and reconstruction metadata, to any device, including electronic device 232.

[0027] FIG. 3 is a flow diagram 300 illustrating a process for rendering an image from a RAW image, generating reconstruction metadata, and creating a reconstructed RAW image using the rendered image and reconstruction metadata. Reconstructing the RAW image 202 requires reconstruction metadata (e.g., reconstruction metadata 204), which may be generated by the camera 130 (e.g., ISP 141) or in an image editing tool (not shown). Here, both the RAW image 202 and a rendered image 302 (e.g., a JPEG image, an sRGB image, etc.) may exist. The RAW image 202 is processed, for example, by the ISP 141 to generate the rendered image 302. Processing the RAW image 202 includes several steps (described below). Each step may involve generating and subsequently storing reconstruction metadata (304). The processor 140 (e.g., a GPU, a CPU, etc.) may generate the reconstruction metadata (304). The generated reconstruction metadata 304 and the rendered image 302 are used to reconstruct 306 the RAW image 202, resulting in a reconstructed RAW image 308. Each step can be performed within a single electronic device, such as the camera 130, or can be performed within multiple electronic devices, as shown in FIG.

[0028] 4 is a schematic block diagram of a reconstruction metadata generator 400 that generates an image reconstruction metadata set. The reconstruction metadata generator 400 may be part of the camera 130. In some embodiments, the image reconstruction metadata set is generated by a processor 411 (e.g., the processor 140, the ISP 141, or some combination thereof) executing machine-readable instructions stored in a memory 401 (e.g., the memory 110). The memory 401 may be transient and / or non-transitory and may include one or both of volatile memory (e.g., SRAM, DRAM, computational RAM, other volatile memory, or any combination thereof) and non-volatile memory (e.g., FLASH, ROM, magnetic media, optical media, other non-volatile memory, or any combination thereof). The memory 401 stores the machine-readable instructions as software. This software may be or include firmware. The software includes a dynamic range compressor 402, a multiband decomposer 404, a subsampler 410, a solver 420, and a multiband decomposer 416. A processor 411 can execute the software to perform the following processes shown in FIG.

[0029] The RAW image 202 is processed by a dynamic range compressor 402 that applies a DRC to the RAW image 202 to reduce the dynamic range of the RAW image 202. In some embodiments, the dynamic range compressor 402 applies the DRC according to a set of DRC parameters 403. The dynamic range compressor 402 may be implemented, for example, by a gamma function of the following formula: RGB'=RGB γ ,0<γ≦1 (1) where RGB γ is the output from DRC 402. In some embodiments, a look-up-table (LUT) can be used to implement any form of non-decreasing monotonic DRC function, for example, of the following equation: RGB'=LUT(RGB) (2) where LUT(RGB) is the output from DRC 402. The resulting image signal is then decomposed into multiple frequency sub-bands in multi-band decomposer 404, including a RAW low-frequency (LF) sub-band 406 and a RAW high-frequency (HF) sub-band 408, which are represented by the following equations: RGB'=LF+HF (3) where LF is the LF sub-band 406 and HF is the HF sub-band 408. In some embodiments, the multiband decomposer 404 uses a low pass filter (LPF) to decompose the image into the RAW LF sub-band 406 and the RAW HF sub-band 408, for example, as follows: LF=LPF(RGB') (4) HF=RGB'-LPF(RGB') (5) Both the DRC parameters 403 and the decomposition parameters 405 are referred to herein as reconstruction metadata (e.g., reconstruction metadata 424) and are stored in memory (e.g., memory 110, 401, 501) and used to reconstruct a RAW image (e.g., RAW image 202) from a rendered image 414 (e.g., which may be in a compressed or uncompressed file format).

[0030] The subsampler 410 sparsely subsamples the RAW LF subband 406 to generate subsampled data 412. The subsampled data 412 may be evenly or unevenly spaced. The size of the subsampled data 412 (e.g., the reconstructed metadata 424) may be further reduced by optional lossless compression (not shown). The subsampled data may be expressed by the following equation: D=LF(R s ,G s ,B s ,x s ,y s ) (6) where D is the subsampled data 412, s represents the sparse subsampled points, and R s , G s , B s , x s , y s are the corresponding R / G / B values ​​and x, y coordinates, respectively. The sparsely subsampled data 412 is referred to herein as reconstruction metadata and may be stored in a memory (e.g., memory 110, 401).

[0031] Multiband decomposer 416 decomposes rendered image 414 using decomposition parameters 405 to generate multiple subbands, including high frequency subband 418. High frequency subband 418 and raw high frequency subband 408 are used by solver 420 to generate reconstruction matrix 422 (shown by equation (7)) using linear estimation techniques known in the art. Reconstruction matrix 422 is expressed as a 3×3 linear matrix obtained from equation (7) below: HF(R,G,B)=M×sHF(R,G,B) (7) where M is the reconstruction matrix 422 and sHF(R,G,B) are the HF subbands 418. The reconstruction matrix 422 may be stored in memory (e.g., in memory 401 along with the subsampled data 412 as reconstruction metadata 424).

[0032] FIG. 5 is a schematic diagram of a RAW image data reconstructor 500 that reconstructs a RAW image from a rendered image using reconstruction metadata. This system may be implemented in one or more aspects of camera 130. In some embodiments, the reconstructed RAW image is generated by a processor 511 (e.g., processor 140, ISP 141, or some combination thereof) executing machine-readable instructions stored in memory 501 (memory 110). Memory 501 may be transient and / or non-transitory and may include one or both of volatile memory (e.g., SRAM, DRAM, computational RAM, other volatile memory, or any combination thereof) and non-volatile memory (e.g., FLASH, ROM, magnetic media, optical media, other non-volatile memory, or any combination thereof). Memory 501 stores the machine-readable instructions as software. This software may be or include firmware. The software includes an inverse dynamic range compressor 508, a multiband decomposer 416, a sparse data interpolator 504, a linear estimator 503, and a multiband composer 506 to perform the following operations shown in FIG.

[0033] A RAW image (e.g., RAW image 202) is reconstructed using the reconstruction metadata (i.e., DRC parameters 403, decomposition parameters 405, subsampling data 412, and reconstruction matrix 422) and a rendered image 414 (e.g., a JPEG image, an sRGB image) to generate a reconstructed RAW image 509. A multiband decomposer 416 decomposes the rendered image 414 into multiple subbands, including HF subbands 418 and LF subbands 502. The decomposition of the rendered image into subbands 418, 502 is performed according to the stored decomposition parameters 405. A sparse data interpolator 504 sparsely interpolates the LF subbands 502 (e.g., via radial basis function interpolation) using the stored subsampling data 412 to generate a reconstructed LF subband 505, which is expressed as the following equation: rLF(R,G,B,x,y)=SparseInterp(LF(Rs ,G s ,B s ,x s ,y s ),sLF(R,G,B,x,y)) (8) where rLF(R,G,B,x,y) is the reconstructed LF subband 505, and LF(R s ,G s ,B s ,x s ,y s ) is the subsampled data 412, and sLF(R,G,B,x,y) is the LF subband 502. The linear estimator 503 linearly estimates the HF subband 418 using the reconstruction matrix 422 and the raw HF subband 408 according to the following equation (9), to generate the reconstructed HF subband 507. rHF(R,G,B) = M × sHF(R,G,B) (9) The multiband composer 506 composes the reconstructed LF sub-band 505 and the reconstructed HF sub-band 507 to generate a reconstructed RAW image 509 according to the following equation: rRGB=rLF+rHF (10) where rRGB is the reconstructed RAW image 509, rLF is the reconstructed LF sub-band 505, and rHF is the reconstructed HF sub-band 507. As shown in FIG. 4 , the dynamic range compressor 402 can apply DRC to the RAW image 202 using the DRC parameters 403 to generate a RAW LF frequency sub-band 406 and a RAW HF frequency sub-band 408. To generate a further reconstructed RAW image 510 in the original camera RAW-RGB domain, the inverse dynamic range compressor 508 applies an inverse DRC (e.g., using a gamma function or a LUT) to the reconstructed RAW image 509 to generate another reconstructed RAW image 510. Applying the inverse DRC is performed according to the DRC parameters 403 from FIG. 4 . For example, the other reconstructed RAW image 510 can then be obtained by one of the following equations (11) and (12):

number

[0034] 4 and 5 may be performed using two or more electronic devices. For example, generating the rendered image and reconstruction metadata may be performed on a first device (e.g., camera 130) and reconstructing the RAW image may be performed on a second device (e.g., laptop 230). For example, camera 130 may generate rendered image 414 and store reconstruction metadata 424 (i.e., DRC parameters 403, decomposition parameters 405, subsampling data 412, reconstruction matrix 422, and RAW high frequency subband 408). Reconstruction metadata 424 may then be transmitted to laptop 230 for use in generating reconstructed RAW image 509 and, in some embodiments, further reconstructed RAW image 510.

[0035] 6 is a flowchart illustrating a method 600 for reconstructing a RAW image. Method 600 may be implemented in one or more aspects of RAW image data reconstructor 500. In some embodiments, method 600 is implemented by processor 511 executing computer-readable instructions stored in memory 501.

[0036] Step 602 includes generating a low-frequency image (e.g., low-frequency sub-band 502) and a high-frequency image (e.g., high-frequency sub-band 418) from an initial image (e.g., rendered image 414) in a file format that may be compressed or uncompressed. In one example of step 602, multiband decomposer 416 decomposes rendered image 414 according to decomposition parameters 405 to obtain images 418 and 502. In one example of step 602, generating the low-frequency image and the high-frequency image further includes decomposing the initial image using a low-pass filter. Step 604 includes linearly estimating the HF image to generate a reconstructed HF image 507. In one example of step 604, solver 420 multiplies high-frequency image 418 by a reconstruction matrix 422.

[0037] Step 606 includes coarsely interpolating the LF image to generate a reconstructed LF image. In one example of step 606, the sparse data interpolator 504 coarsely interpolates the LF image 502 according to the sparse sub-sampled data 412 to obtain a reconstructed LF image 505. Step 608 includes generating a reconstructed RAW image from the reconstructed LF image and the reconstructed HF image. In one example of step 608, the multiband composer 506 combines the reconstructed LF image 505 and the reconstructed HF image 507 to generate a reconstructed RAW image 509.

[0038] In some embodiments, method 600 may include additional or alternative steps. These steps may include applying dynamic range compression (DRC) to the RAW image using a set of dynamic range compression parameters to generate an encoded RAW image. The initial image is derived from the RAW image. For example, dynamic range compressor 402 applies DRC to RAW image 202 to generate an encoded RAW image, which is received by multiband decomposer 404. In such embodiments, method 600 may include step 610, which includes applying inverse DRC to the reconstructed RAW image to generate a reconstructed RAW image. In one example of step 610, inverse dynamic range compressor 508 applies inverse DRC to reconstructed RAW image 509 according to a set of DRC parameters (e.g., DRC parameters 403) to generate a further reconstructed RAW image 510. In some embodiments, method 600 may further include demosaicing RAW image 202 to obtain initial image 414.

[0039] The method 600 may further include generating a RAW low-frequency image and a RAW high-frequency image from the encoded RAW image using the decomposition parameters, and may further include subsampling the RAW low-frequency image to generate subsampled data. For example, the subsampler 410 subsamples the RAW low-frequency image 406 to generate subsampled data 412.

[0040] The method 600 may further include generating a RAW low frequency image 406 from the RAW image 202 and subsampling the RAW low frequency image 406 to generate subsampled data 412. In one example of step 606, sparsely interpolating may include sparsely interpolating the RAW low frequency image 406 according to the subsampling data 412.

[0041] 7 is a flowchart illustrating a method 700 for generating an image reconstruction metadata set. Method 700 may be performed by one or more aspects of reconstruction metadata generator 400. In some embodiments, method 700 is performed by processor 411 executing computer-readable instructions stored in memory 401. Step 702 includes generating a RAW low-frequency image and a RAW high-frequency image from the RAW image or an image derived from the RAW image using decomposition parameters. In one example of step 702, multiband decomposer 404 decomposes RAW image 202 (or an image derived from RAW image 202) according to decomposition parameters 405 to generate a RAW low-frequency image 406 and a RAW high-frequency image 408. In one example of step 702, multiband decomposer 404 uses a low-pass filter as shown in equation (4).

[0042] Step 704 includes subsampling the raw low-frequency image to generate subsampled data. In one example of step 704, a subsampler 410 sparsely subsamples the raw low-frequency image 406 to generate subsampled data 412. The subsampled data 412 may be stored in a memory, such as memory 110 or 401, as reconstruction metadata 424. Step 706 includes filtering the rendered image with decomposition parameters to obtain a high-frequency image. The rendered image is derived from the raw image or an image derived from the raw image. In one example of step 706, a multiband decomposer 416 filters the rendered image 414 to obtain high-frequency sub-bands 418.

[0043] Step 708 includes determining a reconstruction matrix. The product of this reconstruction matrix and the high frequency image is equal to the raw high frequency image. In one example of step 708, solver 420 determines reconstruction matrix 422, which is expressed as a 3×3 linear matrix obtained from equation (7) as described above. The product of reconstruction matrix 422 and high frequency image 418 is equal to the raw high frequency image 408.

[0044] In some embodiments, method 700 may include additional steps before multiband decomposer 404 decomposes RAW image 202 in step 702. In a first example of additional steps, dynamic range compressor 402 applies dynamic range compression (DRC) to RAW image 202 (which may be captured by image sensor 132) using DRC parameters 403 to generate an encoded RAW image (i.e., an image derived from the RAW image). In this example, the DRC parameters are stored in memory (e.g., memory 110, 401, stored as reconstruction metadata 424). In a second example, dynamic range compressor 402 applies DRC using a gamma function, as shown by equation (1), to generate an encoded RAW image (i.e., an image derived from the RAW image). In a third example, the dynamic range compressor 402 applies the DRC using a look-up table (LUT) by implementing any form of non-decreasing monotonic DRC function, as shown by equation (2), to generate an encoded RAW image (i.e., an image derived from the RAW image). In a fourth example, applying the DRC to the RAW image includes gamma encoding the RAW image according to a set of DRC parameters to generate the encoded RAW image (i.e., an image derived from the RAW image).

[0045] In some embodiments, method 700 may include additional or alternative steps. For example, method 700 may further include demosaicing the RAW image to obtain a rendered image.

[0046] 8 is a schematic diagram of a RAW image data reconstructor 800 that reconstructs a RAW image from a rendered image and reconstruction metadata. The RAW image reconstructor 800 includes a reconstruction metadata generator 400 and a RAW image reconstructor 500 communicatively connected to the reconstruction metadata generator 400. The RAW image reconstructor 800 may be provided in any device (e.g., camera 130, laptop 230, smart device 232, etc.) or multiple devices. For example, some of the operations, such as capturing the RAW image 202 and the operations performed by the reconstruction metadata generator 400 (e.g., generating the rendered image 414 and reconstruction metadata 424), may be performed in one device, while other operations, such as reconstructing the RAW image, may be performed in another device. For example, the reconstruction metadata generator 400 may be provided in a first device (e.g., camera 130), and the RAW image reconstructor 500 may be provided in a second device (e.g., laptop 230). Alternatively, they may be provided on the same device.

[0047] As described in the description of the reconstruction metadata generator 400 in Figure 4, the RAW image 202 is input to the reconstruction metadata generator 400, which outputs reconstruction metadata 424. Additionally, the image signal processor 141 may receive the RAW image 202 and output a rendered image 414, as described in Figure 3. The output rendered image 414 and reconstruction metadata 424 may then be transferred to another device or may remain within the same device for input to the RAW image reconstructor 500. The RAW image reconstructor 500 outputs a reconstructed RAW image 509, as described in the description of the RAW image data reconstructor 500 in Figure 5.

[0048] FIG. 9 is a flowchart illustrating a method 900 for reconstructing a RAW image from a rendered image and reconstruction metadata. Method 900 includes steps 902-908. Step 902 may be performed using reconstruction metadata generator 400. Steps 904-908 may be performed using RAW image reconstructor 500. In some embodiments, method 900 is performed by processor 511 executing computer-readable instructions stored in memory 501. Step 902 includes performing method 700 to generate sub-sampled data and a reconstruction matrix. Step 904 includes linearly estimating a reconstructed high-frequency image that provides the high-frequency image multiplied by the reconstruction matrix. The linear estimation of step 904 may include multiplying high-frequency image 418 by reconstruction matrix 422, which, when multiplied by high-frequency image 418, provides RAW high-frequency image 408, according to at least Equation (7). In one example of step 904, the linear estimator 503 linearly estimates the reconstructed high frequency subbands 507 according to equation (9), which gives the HF subbands 418 multiplied by the reconstruction matrix 422. Further, the RAW HF image 408 is obtained by multiplying the HF subbands 418 by the reconstruction matrix 422.

[0049] Step 906 includes sparsely interpolating the low-frequency image using the generated sub-sampled data to generate a reconstructed low-frequency image. In one example of step 906, the sparse interpolator 504 sparsely interpolates the low-frequency image 502 using the sub-sampled data 412 to generate a reconstructed low-frequency image 505. Step 908 includes generating a reconstructed RAW image from the reconstructed low-frequency image and the reconstructed high-frequency image. In a first example of step 908, the multiband composer 506 generates a reconstructed RAW image 509 from the reconstructed low-frequency image 505 and the reconstructed high-frequency image 507. In a second example of step 908, the generating includes combining the reconstructed low-frequency image 505 and the reconstructed high-frequency image 507.

[0050] Method 900 may include additional or alternative steps. For example, method 900 includes reversing the additional steps of method 700 in which the dynamic range compressor 402 applies the dynamic range DRC to the RAW image 202. The additional steps of method 900 include reversing the dynamic range compression of the reconstructed RAW image using the DRC parameters to generate a reconstructed RAW image. In a first example, the inverse dynamic range compressor 508 applies inverse dynamic range compression to the reconstructed RAW image 509 using the DRC parameters 403 to generate a further reconstructed RAW image 510. In a second example, the DRC parameters include coding gamma, and inverting the DRC of the reconstructed RAW image 509 further includes applying inverse gamma correction using the coding gamma. In a third example, the DRC parameters include a lookup table, and inverting the DRC of the reconstructed RAW image further includes using the lookup table. In a fourth example, applying the inverse DRC to the reconstructed RAW image 509 includes gamma encoding the reconstructed RAW image 509 according to a set of DRC parameters.

[0051] Changes may be made in the above method and system without departing from the scope of the present invention. It should therefore be noted that the matter contained in the above description or shown in the accompanying drawings should be interpreted as illustrative and not in a limiting sense. The appended claims are intended to encompass all general and specific features described herein and all descriptions of the scope of the method and system that may literally lie therebetween.

[0052] Combination of features The features described above and in the claims below can be combined in various ways without departing from the scope of the invention. The examples listed below illustrate some possible non-limiting combinations.

[0053] (A1) A method for reconstructing a RAW image, comprising: generating a low frequency image and a high frequency image from the initial image; linearly estimating the high frequency image to generate a reconstructed high frequency image; sparsely interpolating the low frequency image to generate a reconstructed low frequency image; generating a reconstructed RAW image from the reconstructed low-frequency image and the reconstructed high-frequency image; A method comprising:

[0054] (A2) applying dynamic range compression to the RAW image using a set of dynamic range compression (DRC) parameters to generate an encoded RAW image, wherein the initial image is derived from the RAW image; and inverting the DRC of the reconstructed RAW image to generate a further reconstructed RAW image; The embodiment of (A1), further comprising:

[0055] (A3) generating a RAW low-frequency image and a RAW high-frequency image from the encoded RAW image using a set of decomposition parameters, wherein linearly estimating includes multiplying the high-frequency image by a reconstruction matrix that, when multiplied by the high-frequency image, gives the RAW high-frequency image; subsampling the RAW low frequency image to generate subsampled data; The embodiment of (A1) or (A2), further comprising:

[0056] (A4) The embodiment according to any one of (A1) to (A3), further comprising demosaicing the RAW image to obtain the initial image.

[0057] (A5) The embodiment according to any one of (A1) to (A4), wherein inverting the dynamic range compression includes using the set of DRC parameters.

[0058] (A6) The embodiment according to any one of (A1) to (A5), wherein generating the low-frequency image and the high-frequency image from the initial image includes using the set of decomposition parameters.

[0059] (A7) The embodiment according to any one of (A1) to (A6), wherein the sparsely interpolating includes sparsely interpolating the low frequency image according to the sub-sampling data.

[0060] (A8) The embodiment described in any one of (A1) to (A7), wherein generating the low-frequency image and the high-frequency image further includes decomposing the initial image using a low-pass filter according to the set of decomposition parameters.

[0061] (A9) generating a RAW low frequency image from the RAW image; subsampling the RAW low frequency image to generate subsampled image data; The embodiment according to any one of (A1) to (A8), further comprising: Sparsely interpolating includes sparsely interpolating the low frequency image according to the sub-sampled image data.

[0062] (B1) A method for generating an image reconstruction metadata set, comprising: generating a raw low frequency image and a raw high frequency image from the raw image or an image derived from said raw image using the decomposition parameters; subsampling the raw low frequency image to generate subsampled data; obtaining a high frequency image by filtering a rendered image using the decomposition parameters, the rendered image being derived from the raw image; determining a reconstruction matrix whose product with the high frequency image is equal to the raw high frequency image; A method comprising:

[0063] (B2) The embodiment of (B1), further comprising demosaicing the RAW image to obtain the rendered image.

[0064] (B3) applying dynamic range compression (DRC) to the RAW image using a set of DRC parameters to derive an encoded RAW image; The embodiment described in (B1) or (B2), wherein the generating step includes generating the RAW low-frequency image and the RAW high-frequency image from the encoded RAW image.

[0065] (B4) The embodiment according to any one of (B1) to (B3), wherein applying DRC to the RAW image includes gamma encoding the RAW image in accordance with the set of DRC parameters.

[0066] (B5) applying DRC to the RAW image includes using a lookup table; The embodiment according to any one of (B1) to (B4), wherein applying the dynamic range compression to the RAW image includes using the lookup table.

[0067] (B6) The embodiment according to any one of (B1) to (B5), further comprising determining the reconstruction matrix as a matrix that, when multiplied by the high frequency image, gives the RAW high frequency image.

[0068] (C1) A method for reconstructing an image, comprising: generating a RAW low frequency image and a RAW high frequency image from the RAW image or an image derived from the RAW image using the decomposition parameters; subsampling the RAW low frequency image to generate subsampled data; obtaining a high frequency image by filtering a rendered image using the decomposition parameters, the rendered image being derived from the raw image; determining a reconstruction matrix whose product with the high frequency image is equal to the raw high frequency image; generating the sub-sampled data and the reconstruction matrix by sparsely interpolating the low frequency image using the subsampled data to generate a reconstructed low frequency image; linearly estimating the raw high frequency image, which gives the high frequency image multiplied by a reconstruction matrix; generating a reconstructed RAW image from the reconstructed low-frequency image and the reconstructed high-frequency image; A method comprising:

[0069] (C2) The embodiment of (C1), further comprising: reversing the dynamic range compression of the reconstructed RAW image using the set of DRC parameters to generate a reconstructed RAW image.

[0070] (C3) the DRC parameter group includes a coding gamma; The embodiment of (C1) or (C2), wherein reversing the dynamic range compression of the reconstructed RAW image further comprises applying an inverse gamma correction using the coded gamma.

[0071] (C4) the DRC parameter set includes a lookup table; The embodiment according to any one of (C1) to (C3), wherein reversing the dynamic range compression of the reconstructed RAW image further comprises using the lookup table.

[0072] (D1) A processor; a memory communicatively connected to the processor and storing machine-readable instructions that, when executed by the processor, cause the processor to perform any one of methods (A1) to (A9), (B1) to (B6), and (C1) to (C4); A system comprising:

Claims

1. 1. A method for generating an image reconstruction metadata set from a RAW image or an image derived from the RAW image and a rendered image derived from the RAW image, comprising: generating a raw low frequency image and a raw high frequency image from the raw image or an image derived from the raw image using decomposition parameters; subsampling the raw low frequency image to generate subsampled data; obtaining a high frequency image by filtering the rendered image using the decomposition parameters; determining a reconstruction matrix whose product with the high frequency image is equal to the raw high frequency image; A method comprising:

2. applying dynamic range compression (DRC) to the raw image using a set of DRC parameters to derive an encoded raw image; the generating step includes generating the RAW low-frequency image and the RAW high-frequency image from the encoded RAW image; The method of claim 1.

3. The method of claim 2 , wherein applying DRC to the RAW image comprises gamma encoding the RAW image in accordance with the set of DRC parameters.

4. The method of claim 2 , wherein applying DRC to the RAW image comprises using a look-up table.

5. 1. A method of reconstructing a RAW image from a rendered image derived from the RAW image, comprising: receiving subsampling data and a reconstruction matrix, the subsampling data representing a subsampled RAW low frequency image generated from the RAW image, and the reconstruction matrix determined such that a product of the reconstruction matrix and a high frequency image generated from the rendering image is equal to a RAW high frequency image generated from the RAW image; sparsely interpolating a low frequency image generated from the rendered image using the sub-sampled data to generate a reconstructed RAW low frequency image; generating a reconstructed raw high-frequency image as a product of a high-frequency image generated from the rendering image and the reconstruction matrix; generating a reconstructed raw image from the reconstructed raw low-frequency image and the reconstructed raw high-frequency image; A method comprising:

6. The method of claim 5 , wherein the low frequency image and the high frequency image are generated by decomposing the rendered image with a low pass filter according to a set of decomposition parameters.

7. The method of claim 5 , further comprising: reversing the dynamic range compression of the reconstructed RAW image using the DRC parameters to generate a further reconstructed RAW image.

8. a processor; a memory communicatively coupled to said processor and storing machine-readable instructions that, when executed by said processor, cause said processor to perform the method of any one of claims 1 to 7; A system comprising:

Images