Original image data reconstruction system and method

Abstract

A method of reconstructing an original image includes: generating a low-frequency image and a high-frequency image from an initial image; performing linear estimation on the high-frequency image to generate a reconstructed high-frequency image; performing sparse interpolation on the low-frequency image to generate a reconstructed low-frequency image; a reconstructed original image is generated from the reconstructed low frequency image and the reconstructed high frequency image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the priority benefit of the following applications: European Patent Application 22166487.3 filed on April 4, 2022 (reference number: D21137EP) and U.S. Provisional Patent Application No. 63 / 326,987 (reference number: D21137USP1), each of which is incorporated herein by reference in its entirety. Background Art

[0003] Many consumer electronic products contain at least one camera. These products include tablets, mobile phones, and smart watches. In these products, as well as in digital cameras themselves, the camera includes an image sensor having many pixels arranged in a pixel array. The image sensor, image signal processor (ISP), or image editing tool can adjust the captured raw image to more accurately depict an image similar to what the human eye sees. Many of the operations performed by the ISP or editing tool are non-linear, even spatially varying, and require significant computing power to invert (if possible at all). The inventors recognized a need for improved raw image reconstruction techniques.

[0004] Mahmoud Afifi: “Image color correction, enhancement, and editing,” Arxiv.org, Cornell University Library, New York 14853, July 28, 2021, XP091018402, discloses methods and approaches for image color correction, color enhancement, and color editing. The color correction problem is studied from the perspective of the camera’s image signal processor (ISP). A camera’s ISP is a piece of hardware that applies a series of in-camera image processing and color processing steps, many of which are inherently nonlinear, to render the initial sensor image into a final photographic representation saved in the 8-bit standard RGB (sRGB) color space. Since white balancing (WB) is one of the primary procedures used by the ISP for color correction, two different ISP white balancing methods are proposed. Another scenario for correcting and editing image color is discussed, where a set of methods are introduced to correct and edit the white balance settings of an image that has been inappropriately white balanced by the ISP. Then, another factor that has a significant impact on the quality of camera-rendered color is explored, where two different methods are outlined to correct exposure errors in camera-rendered images. Finally, automatic color editing and processing after capture is discussed. In particular, automatic image recoloring methods are proposed to generate different realistic versions of the same camera-rendered image using new colors.

[0005] Abhijith Punnappurath et al., “Spatially Aware Metadata for RawReconstruction,” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, January 3, 2021, pages 218–226, XP033926466, discloses a raw reconstruction method based on spatially aware metadata. A camera sensor captures a raw RGB image, which is then processed into a standard RGB (sRGB) image through a series of onboard operations performed by the camera’s image signal processor (ISP). Among these processing steps, local tone mapping is one of the most important operations used to enhance the overall appearance of the final rendered sRGB image. For some applications, it is often necessary to un-render or process the sRGB image back to its original raw RGB values. This “raw reconstruction” is a challenging task because the operations performed by the ISP, including local tone mapping, are non-linear and difficult to invert.

[0006] Existing raw reconstruction methods (which store specific metadata at capture time to achieve raw restoration) ignore local tone mapping and assume a global transformation between raw RGB and sRGB color spaces. The disclosed raw reconstruction method based on spatially aware metadata is robust to local tone mapping and has significantly higher raw reconstruction accuracy (6dB average PSNR improvement) than existing raw reconstruction methods. The disclosed method only requires 0.2% samples of the full-size image as metadata, the computational overhead at capture time is negligible, and can be easily integrated into modern ISPs.

[0007] Nguyen Rang et al., “RAW Image Reconstruction Using a Self-Contained sRGB-JPEG Image with Only 64KB Overhead”, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, June 27, 2016, pp. 1655-1663, XP033021343, discloses a method for reconstructing a RAW image from an sRGB-JPEG image. Most camera images are saved as 8-bit standard RGB (sRGB) compressed JPEGs. Even when JPEG compression is set to its highest quality, the encoded sRGB images are significantly processed in terms of color and tonal manipulation. This makes sRGB-JPEG images unsuitable for many computer vision tasks that assume a direct relationship between pixel values ​​and incident light. For such applications, the RAW image format is preferred because RAW represents a minimally processed, sensor-specific RGB image with a higher dynamic range that is linearly related to the scene radiance. However, the disadvantage of RAW images is that they require a lot of storage space and are not well supported by many imaging applications. To address this issue, a method is proposed as follows: necessary metadata in sRGB images is encoded to reconstruct high-quality RAW images. The disclosed method does not require camera calibration and can reconstruct the original RAW to within 0.3% error and requires only 64KB of additional data overhead. More importantly, the output is a completely independent 100% complementary sRGB-JPEG file that can be used as is without affecting any existing image workflow - the RAW image can be extracted when needed and ignored otherwise. Summary of the invention

[0008] The invention is defined by the independent claims. The dependent claims relate to optional features of some embodiments. According to one embodiment, a method for reconstructing original image data comprises: generating a low-frequency image and a high-frequency image from an initial image; performing linear estimation on the high-frequency image to generate a reconstructed high-frequency image; performing sparse interpolation on the low-frequency image to generate a reconstructed low-frequency image; and generating a reconstructed original image from the reconstructed low-frequency image and the reconstructed high-frequency image.

[0009] According to one embodiment, a method for generating an image reconstruction metadata set includes: using decomposition parameters to generate an original low-frequency image and an original high-frequency image from an original image or an image derived from the original image; subsampling the original low-frequency image to generate subsampled data; filtering a rendered image using the decomposition parameters to produce a high-frequency image, the rendered image being derived from the original image; and determining a reconstruction matrix, the product of which and the high-frequency image is equal to the original high-frequency image.

[0010] According to one embodiment, a method for reconstructing an image includes: generating an original low-frequency image and an original high-frequency image from an original image or an image derived from the original image by using decomposition parameters, subsampling the original low-frequency image to generate subsampled data, and filtering a rendered image using the decomposition parameters to produce a high-frequency image, wherein the rendered image is derived from the original image; and determining a reconstruction matrix, wherein the product of the reconstruction matrix and the high-frequency image is equal to the original high-frequency image; sparsely interpolating the low-frequency image using the subsampled data to generate a reconstructed low-frequency image; linearly estimating the original high-frequency image, which is produced by multiplying the high-frequency image by the reconstruction matrix; and generating a reconstructed original image from the reconstructed low-frequency image and the reconstructed high-frequency image.

[0011] According to one embodiment, a system includes: a processor; a memory, the memory being communicatively coupled to the processor and storing machine-readable instructions, which, when executed by the processor, cause the processor to: generate a low-frequency image and a high-frequency image from an initial image; perform linear estimation on the high-frequency image to generate a reconstructed high-frequency image; perform sparse interpolation on the low-frequency image to generate a reconstructed low-frequency image; and generate a reconstructed original image from the reconstructed low-frequency image and the reconstructed high-frequency image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A situation in which a camera according to an embodiment images a scene having a high dynamic range of brightness is depicted.

[0013] Figure 2 is a diagram showing that the camera according to the embodiment Figure 1 A flowchart of sending the rendered image together with the generated reconstruction metadata to other electronic devices.

[0014] Figure 3 is a flow chart illustrating a process of reconstructing an original image using a rendered image and generated reconstruction metadata according to an embodiment.

[0015] Figure 4 is a schematic block diagram of a reconstruction metadata generator for generating an image reconstruction metadata set according to an embodiment.

[0016] Figure 5 According to the use of the embodiment Figure 4A schematic diagram of an original image data reconstructor for reconstructing an original image from a rendered image.

[0017] Figure 6 is a flowchart illustrating a method of reconstructing an original image from an initial image according to an embodiment.

[0018] Figure 7 is a flow chart illustrating a method of generating an image reconstruction metadata set according to an embodiment.

[0019] Figure 8 According to an embodiment, Figure 4 Refactored metadata generator and Figure 5 Schematic diagram of a raw image data reconstructor of a raw image data reconstructor.

[0020] Fig. 9 is a flow chart illustrating a method of reconstructing an original image from a rendered image and reconstruction metadata. DETAILED DESCRIPTION

[0021] In modern digital cameras and smart devices (such as smartphones), image sensors capture real scenes and convert the captured real scenes into raw RGB signals for the camera, also called scene-referenced data. An image signal processor (ISP) (hereinafter used interchangeably with "processor") processes the raw RGB signals (hereinafter used interchangeably with "raw image") to obtain a final output signal, which is usually called display reference (interchangeably used with "rendered image" and "initial image") and can be in a standard color space, such as sRGB, P3, Bt2020, etc. The final output signal is usually compressed (but may not be compressed) and stored in some standard image or video file formats, such as JPEG, MPEG, AVC, HEVC, etc.

[0022] In order to convert the raw RGB signal into a realistic scene (e.g., mimicking a scene that a human eye would see), the ISP typically performs a series of computationally intensive image operations, such as black level correction, lens shading correction, white balance, color correction, global / local tone mapping, denoising, sharpening, and / or other forms of image or video enhancement operations. Camera ISPs are typically proprietary and have different parameters established by different camera manufacturers for tuning preferences. As a result, the perception of color, contrast, sharpness, noise, details, etc., output by a camera capturing the same scene with the same field of view may sometimes be completely different from that of other cameras. Some ISP operations are nonlinear and even spatially varying, making them difficult to invert to reconstruct the original RGB signal. However, in some applications, it is desirable to invert the ISP's processing back to the original signal domain. For example, digital cameras are often adjusted for visually pleasing or aesthetic purposes, which may exaggerate color saturation and enhance memory color enhancement. In order to achieve accurate color reproduction of scene reference images / data, it is very important to remove redundant ISP photo-finishing operations (such as saturation enhancement, selective color enhancement, etc.). Another example is that sometimes users may need to have consistent scene reference images / data even when capturing real scenes from different electronic devices. Once users obtain the camera raw images, users can re-render the raw images to ensure consistent output according to their preferences.

[0023] Other applications include image operations that operate in linear space, such as color / white balance post correction. Cameras often use an in-camera automatic white balance (AWB) algorithm to estimate the scene illumination at the time of capture. However, the AWB algorithm does not always estimate the scene illumination correctly, and therefore may apply incorrect white balance gains on the raw image, resulting in unsightly color casts in the final output image. While post correction of these color errors is simple in the linear raw image domain, correcting them in the final output image domain is very difficult due to the non-linear and spatially variable operations required for the final output.

[0024] Camera raw image reconstruction may also find important use cases in recent deep learning training data augmentation and expansion, which often requires simulating the same scene under different lighting and different image operations. Most simulations need to be performed in linear raw space to obtain better results. Although users may save raw images as the final output at the time of capture (e.g., JPEG / HEIC images) for subsequent processing, the size of raw images is usually large, which is not conducive to storage and communication. Therefore, in order to inverse the ISP rendering and reconstruct the scene reference image / data directly from the output image / video, generating information (reconstruction metadata) (e.g., small size data) when performing operations during rendering is not only very important for in-camera image / video processing and offline image / video processing / editing tools, but also for image / video storage and communication. This embodiment discusses JPEG images in sRGB space, however, these are only for illustrative purposes. This embodiment is applicable to any color space and any existing image / video file format. For example, it may be used for iPhones that output HEIC images in P3 color space. In addition, in the case of HDR capture, it can also be extended to R.2020PQ or HLG.

[0025] Figure 1 Camera 130 is depicted imaging scene 120. Scene 120 includes person 121. Camera 130 includes an imaging lens (not shown), image sensor 132, memory 110, and processor 140 communicatively coupled to image sensor 132. Memory 110 may be transient and / or non-transitory and may include one or both of volatile memory (e.g., SRAM, DRAM, computational RAM, other volatile memory, or any combination thereof) and non-volatile memory (e.g., FLASH, ROM, magnetic media, optical media, other non-volatile memory, or any combination thereof). Part or all of memory 110 may be integrated into processor 140. Processor 140 may be a dedicated image signal processor or a general purpose processor (e.g., a GPU, DSP, microprocessor, etc.). Image sensor 132 includes pixel array 134A, which may have a color filter array (CFA) 136 thereon. Pixel array 134A includes a plurality of pixels 134, which are not shown in the figure for clarity of illustration. Figure 1 1. Each color filter of CFA 136 is aligned with a corresponding pixel 134 of pixel array 134A. Imaging lens images scene 120 onto image sensor 132. Image sensor 132 also includes circuitry 138 that includes at least one analog-to-digital converter.

[0026] Each pixel has a corresponding pixel charge corresponding to a corresponding intensity of light imaged onto pixel array 134A from scene 120. Circuitry 138 converts each pixel charge into a corresponding one of a plurality of pixel values ​​194 (e.g., raw image 202, see Figure 2 ), which pixel values ​​are stored in the memory 110. The camera 130 may include a display 139 configured to display the pixel values ​​194 (e.g., the rendered image 204) as a first captured image 296. The display 139 may display the pixel values ​​194 as a real-time preview. To generate the rendered image 204, the processor 140 may perform a series of operations (in the example of FIG. 1 ) on the original image according to the instructions stored in the memory 110. Figure 6 204, and for reconstructing the original image from the rendered image, as described below. The ISP may generate a rendered image 141. Generating a rendered image, reconstructing metadata, and reconstructing an original image are discussed below.

[0027] Figure 2 2 is a flow chart 200 showing that the camera 130 transmits the original image and / or the rendered image and the reconstruction metadata 204 to another electronic device 230 (e.g., a laptop, a desktop, etc.). The laptop 230 may use the original image 202 to generate the rendered image and the reconstruction metadata 204. For example, the user of the laptop 230 edits the original image 202 to produce an aesthetically pleasing rendered image. Alternatively, the user of the laptop 230 wants to change the rendered image, and since non-linear operations are performed when generating the rendered image, changing the image may require reconstructing the original image 202 to start over. In addition, the laptop 230 may transmit the original image 202 (or the reconstructed original image) or the rendered image and the reconstruction metadata 204 to another electronic device 232. Likewise, the electronic device 232 may use the original image 202 to produce a rendered image, use the reconstructed original image to produce another rendered image, or use the rendered image and the reconstruction metadata 204 to reconstruct the original image.

[0028] Figure 2 For illustration purposes only, and not intended to limit the present embodiment. For example, the camera 130 may transmit the original image 202 and the rendered image and the reconstruction metadata to any device, including the electronic device 232 .

[0029] Figure 3300 is a flowchart illustrating a process of rendering an image from an original image, generating reconstruction metadata, and using the rendered image and the reconstruction metadata to produce a reconstructed original image. The reconstructed original image 202 requires reconstruction metadata (e.g., reconstruction metadata 204), which may be generated by the camera 130 (e.g., the ISP 141) or generated in an image editing tool (not shown), where both the original image 202 and the rendered image 302 (e.g., a JPEG, sRGB image, etc.) may exist simultaneously. For example, the original image 202 is processed by the ISP 141 to generate the rendered image 302. Processing the original image 202 includes several steps (described below), and each step may result in the generation and then storage of reconstruction metadata 304. The processor 140 (e.g., a GPU, a CPU, etc.) may generate the reconstruction metadata 304. The generated reconstruction metadata 304 and the rendered image 302 are used to reconstruct 306 the original image 202, thereby obtaining a reconstructed original image 308. Each step may be performed within a single electronic device (e.g., the camera 130) or multiple electronic devices, such as Figure 2 shown.

[0030] Figure 4 is a schematic block diagram of a reconstruction metadata generator 400 that generates an image reconstruction metadata set. The reconstruction metadata generator 400 may be part of the camera 130. In an embodiment, the image reconstruction metadata set is generated by a processor 411 (e.g., the processor 140, the ISP 141, or some combination thereof) executing machine-readable instructions stored in a memory 401 (e.g., the memory 110). The memory 401 may be transient and / or non-transient, and may include one or both of a volatile memory (e.g., SRAM, DRAM, computational RAM, other volatile memory, or any combination thereof) and a non-volatile memory (e.g., FLASH, ROM, magnetic media, optical media, other non-volatile memory, or any combination thereof). The memory 401 stores the machine-readable instructions as software, which may be firmware or include firmware. The software includes a dynamic range compressor 402, a multi-band decomposer 404, a subsampler 410, a solver 420, and a multi-band decomposer 416. The processor 411 may execute the software to perform Figure 4 The following operations are discussed in .

[0031] The original image 202 is processed by a dynamic range compressor 402, which applies DRC to the original image 202 to reduce the dynamic range of the original image 202. In an embodiment, the dynamic range compressor 402 applies DRC according to DRC parameters 403. The dynamic range compressor 402 may be implemented by a gamma function, such as

[0032] RGB′=RGB γ , 0<γ≤1 (1)

[0033] Among them, RGB γ is the output from DRC 402. In an embodiment, a lookup table (LUT) may be used to implement any form of non-decreasing monotonic DRC function, such as

[0034] RGB′=LUT(RGB) (2)

[0035] Where LUT(RGB) is the output from DRC 402. The resulting image signal is then decomposed into a plurality of frequency sub-bands at a multi-band decomposer 404, including an original low frequency (LF) sub-band 406 and an original high frequency (HF) sub-band 408, represented as

[0036] RGB′=LF+HF (3)

[0037] Where LF is the LF subband 406 and HF is the HF subband 408. In an embodiment, the multi-band decomposer 404 decomposes the image into the original LF subband 406 and the original HF subband 408 using a low pass filter (LPF), for example

[0038] LF=LPF(RGB′) (4)

[0039] HF=RGB′-LPF(RGB′) (5)

[0040] Both the DRC parameters 403 and the decomposition parameters 405 are referred to herein as reconstruction metadata (e.g., reconstruction metadata 424), stored in a memory (e.g., memory 110, 401, 501), and used to reconstruct an original image (e.g., original image 202) from a rendered image 414 (e.g., which may be in a compressed file format or an uncompressed file format).

[0041] The subsampler 410 performs sparse subsampling on the original LF subband 406 to generate subsampled data 412. The subsampled data 412 may be evenly spaced or unevenly spaced. The size of the subsampled data 412 (e.g., the reconstructed metadata 424) may be further reduced by optional lossless compression (not shown). The subsampled data may be represented by the following formula:

[0042] D=LF(R s , G s , B s , x s ,y s ). (6)

[0043] Where D is the sub-sampled data 412, s represents the sparse sub-sampling point, R s , G s , B s ,xs , t s The corresponding R / G / B values ​​and x,y coordinates, respectively. The sparse sub-sampled data 412 is referred to herein as reconstruction metadata and may be stored in a memory (eg, memory 110, 424).

[0044] The multi-band decomposer 416 decomposes the rendered image 414 using the decomposition parameters 405 to generate a plurality of sub-bands, including a high-frequency sub-band 418. The solver 420 uses the high-frequency sub-band 418 and the original high-frequency sub-band 408 to generate a reconstruction matrix 422 (as shown in equation (7)) using linear estimation techniques known in the art. The reconstruction matrix 422 is represented as a 3×3 linear matrix, which is derived from equation (7).

[0045] HF(R,G,B)=M*sHF(R,G,B) (7)

[0046] Where M is the reconstruction matrix 422 and sHF(R, G, B) is the HF subband 418. The reconstruction matrix 422 may be stored in a memory (eg, stored in the memory 401 together with the subsampled data 412 as reconstruction metadata 424).

[0047] Figure 5 is a schematic diagram of a raw image data reconstructor 500 that reconstructs a raw image from a rendered image using reconstruction metadata. The system may be implemented within one or more aspects of the camera 130. In an embodiment, the reconstructed raw image is generated by a processor 511 (e.g., the processor 140, the ISP 141, or some combination thereof) executing machine-readable instructions stored in a memory 501 (memory 110). The memory 501 may be transient and / or non-transient, and may include one or both of a volatile memory (e.g., SRAM, DRAM, computational RAM, other volatile memory, or any combination thereof) and a non-volatile memory (e.g., FLASH, ROM, magnetic media, optical media, other non-volatile memory, or any combination thereof). The memory 501 stores machine-readable instruction software, which may be firmware, or include firmware. The software includes an inverse dynamic range compressor 508, a multi-band decomposer 416, a sparse data interpolator 504, a linear estimator 503, and a multi-band decomposer 506 to perform Figure 5 The following operations are discussed in .

[0048] The original image (e.g., original image 202) is reconstructed using the reconstruction metadata (i.e., DRC parameters 403, decomposition parameters 405, sub-sampling data 412, and reconstruction matrix 422) and the rendered image 414 (e.g., a JPEG sRGB image) to generate a reconstructed original image 509. The multi-band decomposer 416 decomposes the rendered image 414 into a plurality of sub-bands, including an HF sub-band 418 and an LF sub-band 502. The rendered image is decomposed into the sub-bands 418, 502 according to the stored decomposition parameters 405. The sparse data interpolator 504 performs sparse interpolation (e.g., by radial basis function interpolation) on the LF sub-band 502 using the stored sub-sampling data 412 to generate a reconstructed LF sub-band 505, represented as

[0049] rLF(R,G,B,x,y)=SparseInterp(LF(R s ,G s , B s , x s ,y s ), sLF(R,G,B,x,y)). (8)

[0050] Where rLF(R, G, B, x, y) is the reconstructed LF subband 505, LF(R s , G s , B s ,x s ,y s ) is the sub-sampled data 412, and sLF(R, G, B, x, y) is the LF sub-band 502. According to equation (9), the linear estimator 503 uses the reconstruction matrix 422 and the original HF sub-band 408 to perform linear estimation on the HF sub-band 418 to generate a reconstructed HF sub-band 507.

[0051] rHF(R,G,B)=M*sHF(R,G,B) (9)

[0052] The multi-band synthesizer 506 synthesizes the reconstructed LF sub-band 505 and the reconstructed HF sub-band 507 according to the following formula to generate a reconstructed original image 509,

[0053] rRGB=rLF+rHF, (10)

[0054] Where rRGB is the reconstructed original image 509, rLF is the reconstructed LF subband 505, and rHF is the reconstructed HF subband 507. Figure 4As shown, the dynamic range compressor 402 may apply DRC to the raw image 202 to generate raw LF and HF frequency sub-bands 406, 408 using DRC parameters 403. To generate a further reconstructed raw image 510 in the original camera raw RGB domain, the inverse dynamic range compressor 508 applies an inverse DRC to the reconstructed raw image 509 (e.g., using a gamma function or LUT) to generate a further reconstructed raw image 510. Figure 4 As can be seen from the figure, the inverse DRC is applied according to the DRC parameters 403. For example, the further reconstructed original image 510 can then be obtained by one of equations (11) and (12),

[0055]

[0056] rRGB′=LUT -1 (rRGB) (12)

[0057] Here, rRGB′ is the original image 510 that is further reconstructed.

[0058] In an embodiment, more than one electronic device may be used to perform Figure 4 and Figure 5 4. For example, a rendered image and reconstruction metadata are generated on a first device (e.g., camera 130), and an original image can be reconstructed on a second device (e.g., laptop 230). For example, camera 130 can generate rendered image 414 and store reconstruction metadata 424 (i.e., DRC parameters 403, decomposition parameters 405, subsampling data 412, reconstruction matrix 422, and original high frequency subbands 408). Reconstruction metadata 424 can then be transmitted to laptop 230 for generating a reconstructed original image 509, and in an embodiment, a further reconstructed original image 510.

[0059] Figure 6 is a flow chart illustrating a method 600 for reconstructing a raw image. The method 600 may be implemented within one or more aspects of the raw image data reconstructor 500. In an embodiment, the method 600 is implemented by the processor 511 executing computer readable instructions stored in the memory 501.

[0060] Step 602 includes generating a low frequency image (e.g., low frequency sub-band 502) and a high frequency image (e.g., high frequency sub-band 418) from an initial image (e.g., rendered image 414), which is a file format that can be compressed or uncompressed. In one example of step 602, multi-band decomposer 416 decomposes rendered image 414 according to decomposition parameters 405 to produce images 418 and 502. In one example of step 602, generating the low frequency image and the high frequency image also includes decomposing the initial image using a low pass filter. Step 604 includes linearly estimating the HF image to generate a reconstructed HF image 522. In one example of step 604, solver 420 multiplies the high frequency image 418 by reconstruction matrix 422.

[0061] Step 606 includes sparsely interpolating the LF image to generate a reconstructed LF image. In one example of step 606, the sparse data interpolator 504 performs sparse interpolation on the LF image 502 based on the sparse sub-sampled data 412 to generate a reconstructed LF image 505. Step 608 includes generating a reconstructed original image from the reconstructed LF image and the reconstructed HF image. In one example of step 608, the multi-band synthesizer 506 synthesizes the reconstructed LF image 505 and the reconstructed HF image 507 to generate a reconstructed original image 509.

[0062] In an embodiment, the method 600 may include additional or alternative steps including applying dynamic range compression to the original image using dynamic range compression (DRC) parameters to generate an encoded original image, the original image being derived from the original image. For example, the dynamic range compressor 402 applies the DRC to the original image 202 to generate an encoded original image, which is received by the multi-band decomposer 404. In such an embodiment, the method 600 may include a step 610, which includes applying an inverse DRC to the reconstructed original image to generate the reconstructed original image. In one example of step 610, the inverse dynamic range compressor 508 applies the inverse DRC to the reconstructed original image 509 according to the DRC parameters (e.g., the DRC parameters 403) to generate a further reconstructed original image 510. In an embodiment, the method 600 may also include demosaicing the original image 202 to produce the initial image 414.

[0063] The method 600 may also include generating an original low frequency image and an original high frequency image from the encoded original image using the decomposition parameters, and further includes subsampling the original low frequency image to generate subsampled data. For example, the subsampler 410 subsamples the original low frequency 406 to generate subsampled data 412.

[0064] The method 600 may further include generating an original low frequency image 406 from the original image 202 and subsampling the original low frequency image 406 to generate subsampled data 412 . In the example of step 606 , the sparse interpolation includes performing sparse interpolation on the original low frequency image 406 according to the subsampled data 412 .

[0065] Figure 7 is a flow chart illustrating a method 700 for generating an image reconstruction metadata set. The method 700 may be implemented within one or more aspects of the reconstruction metadata generator 400. In an embodiment, the method 700 is implemented by the processor 411 executing computer-readable instructions stored in the memory 401. Step 702 includes generating an original low-frequency image and an original high-frequency image from an original image or an image derived from the original image using decomposition parameters. In an example of step 702, the multi-band decomposer 404 decomposes the original image 202 (or an image derived from the original image) according to the decomposition parameters 405 to generate an original low-frequency image 406 and an original high-frequency image 408. In an example of step 702, the multi-band decomposer 404 uses a low-pass filter, as shown in equation (4).

[0066] Step 704 includes subsampling the original low-frequency image to generate subsampled data. In one example of step 704, the subsampler 410 performs sparse subsampling on the original low-frequency image 406 to generate subsampled data 412. The subsampled data 412 can be stored in a memory, such as the memory 110, 401, as reconstruction metadata 424. Step 706 includes filtering the rendered image using the decomposition parameters to generate a high-frequency image. The rendered image is derived from the original image, or is derived from an image derived from the original image. In one example of step 706, the multi-band decomposer 416 filters the rendered image 414 to generate high-frequency subbands 418.

[0067] Step 708 includes determining a reconstruction matrix, wherein the product of the reconstruction matrix and the high frequency image is equal to the original high frequency image. In one example of step 708, the solver 420 determines a reconstruction matrix 422, wherein the reconstruction matrix 422 is represented as a 3×3 linear matrix, which is derived from equation (7), as described above. The product of the reconstruction matrix 422 and the high frequency image 418 is equal to the original high frequency image 408.

[0068] In an embodiment, the method 700 may include additional steps before the multi-band decomposer 404 decomposes the original image 202 in step 702. In a first example of additional steps, the dynamic range compressor 402 applies dynamic range compression (DRC) to the original image 202 (which may be captured by the image sensor 132) using DRC parameters 403 to generate an encoded original image (i.e., an image derived therefrom). In this example, the DRC parameters are stored in a memory (e.g., the memory 110, 401, stored as reconstruction metadata 424). In a second example, the dynamic range compressor 402 applies DRC to generate an encoded original image (i.e., an image derived therefrom) using a gamma function (as shown in equation (1)). In a third example, the dynamic range compressor 402 applies DRC using a lookup table (LUT) to generate an encoded original image (i.e., an image derived therefrom) by implementing any form of non-decreasing monotonic DRC function, as shown in equation (2). In a fourth example, applying the DRC to the raw image includes gamma encoding the raw image to generate an encoded raw image (ie, an image derived therefrom) according to the DRC parameters.

[0069] In an embodiment, the method 700 may include additional or alternative steps. For example, the method 700 may also include demosaicing the original image to generate a rendered image.

[0070] Figure 8 Schematic diagram of a raw image data reconstructor 800 for reconstructing a raw image from a rendered image and reconstructed metadata. The raw image reconstructor 800 includes a reconstructed metadata generator 400 and a raw image reconstructor 500 communicatively coupled thereto. The raw image reconstructor 800 may be located in any device (e.g., camera 130, laptop 230, smart device 232, etc.) or multiple devices. For example, some operations, such as capturing the raw image 202 and the operations performed by the reconstructed metadata generator 400 (e.g., generating the rendered image 414 and reconstructed metadata 424) may be performed on one device, while other operations (e.g., reconstructing the raw image) may be performed on another device. For example, the reconstructed metadata generator 400 may be located on a first device (e.g., camera 130) and the raw image reconstructor 500 may be located on a second device (e.g., laptop 230), or they may be located on the same device.

[0071] The original image 202 is input to the reconstruction metadata generator 400, which outputs the reconstruction metadata 424, as described in the description of the reconstruction metadata generator 400 ( Figure 4 In addition, the image signal processor 141 can receive the original image 202 and output the rendered image 414, such as Figure 3The output rendered image 414 and reconstruction metadata 424 can then be passed to another device, or retained within the same device, to be input to the raw image reconstructor 500, where the reconstructed raw image 509 is output, as described in the description of the raw image data reconstructor 500 ( Figure 5 ) as discussed in .

[0072] Fig. 9 is a flow chart showing a method 900 for reconstructing an original image from a rendered image and reconstruction metadata. The method 900 includes steps 902 to 908. Step 902 may be implemented using the reconstruction metadata generator 400. Steps 904-908 may be implemented using the original image reconstructor 500. In an embodiment, the method 900 is implemented by the processor 511 executing computer-readable instructions stored in the memory 501. Step 902 includes executing the method 700 to generate sub-sampled data and a reconstruction matrix. Step 904 includes linearly estimating a reconstructed high-frequency image, which yields the high-frequency image multiplied by the reconstruction matrix. The linear estimation of step 904 may include multiplying the high-frequency image 418 by the reconstruction matrix 422, and when the high-frequency image 418 is multiplied by the reconstruction matrix 422, the original high-frequency image 408 is obtained at least according to equation (7). In the example of step 904, the linear estimator 503 linearly estimates the reconstructed high-frequency sub-band 507 according to equation (9), which yields the HF sub-band 418 multiplied by the reconstruction matrix 422. Furthermore, the HF sub-band 418 is multiplied by the reconstruction matrix 422 to obtain the original HF image 408 .

[0073] Step 906 includes performing sparse interpolation on the low-frequency image using the generated sub-sampled data to generate a reconstructed low-frequency image. In one example of step 906, the sparse interpolator 504 performs sparse interpolation on the low-frequency image 502 using the sub-sampled data 412 to generate a reconstructed low-frequency image 505. Step 908 includes generating a reconstructed original image from the reconstructed low-frequency image and the reconstructed high-frequency image. In a first example of step 908, the multi-band synthesizer 506 generates a reconstructed original image 509 from the reconstructed low-frequency image 505 and the reconstructed high-frequency image 507. In a second example of step 908, the generating includes synthesizing the reconstructed low-frequency image 505 and the reconstructed high-frequency image.

[0074] Method 900 may include additional or alternative steps. For example, method 900 includes performing an inverse operation for the additional step of applying a dynamic range DRC to the original image 202 by the dynamic range compressor 402 of method 700. The additional step of method 900 includes performing an inverse dynamic range compression of the reconstructed original image using DRC parameters to generate a reconstructed original image. In a first example, the inverse dynamic range compressor 508 applies inverse dynamic range compression to the reconstructed original image 509 using the DRC parameters 403 to generate a further reconstructed original image 510. In a second example, the DRC parameters include an encoding gamma, and the inverse DRC of the reconstructed original image 509 also includes applying an inverse gamma correction using the encoding gamma. In a third example, the DRC parameters include a lookup table, and performing inverse DRC on the reconstructed original image also includes using the lookup table. In a fourth example, applying inverse DRC to the reconstructed original image 509 includes gamma encoding the reconstructed original image 509 according to the DRC parameters.

[0075] Changes may be made to the above-described methods and systems without departing from the scope of the present invention. It should therefore be noted that the matter contained in the above description or shown in the accompanying drawings is to be interpreted as illustrative and not restrictive. The following claims are intended to cover all generic and specific features described herein, as well as all statements of the scope of the present methods and systems, which, by definition, can be said to fall therein.

[0076] Combination of features

[0077] The above features and the features of the following claims may be combined in various ways without departing from the scope of the present invention. The following examples illustrate some possible, non-limiting combinations:

[0078] (A1) A method for reconstructing an original image, comprising: generating a low-frequency image and a high-frequency image from an initial image; performing linear estimation on the high-frequency image to generate a reconstructed high-frequency image; performing sparse interpolation on the low-frequency image to generate a reconstructed low-frequency image; and generating a reconstructed original image from the reconstructed low-frequency image and the reconstructed high-frequency image.

[0079] (A2) In the embodiment indicated by (A1), further comprising applying dynamic range compression (DRC) parameters to the original image to generate the encoded original image, the original image being derived from the original image; and

[0080] The DRC of the reconstructed original image is inverted to generate a further reconstructed original image.

[0081] (A3) In the embodiment indicated by (A1) or (A2), it also includes using decomposition parameters to generate an original low-frequency image and an original high-frequency image from the encoded original image, wherein the linear estimation includes multiplying the high-frequency image by a reconstruction matrix, and when the reconstruction matrix is ​​multiplied by the high-frequency image, the original high-frequency image is obtained; and the original low-frequency image is sub-sampled to generate sub-sampled data.

[0082] (A4) In any of the embodiments indicated by any of (A1)-(A3), it also includes demosaicing the original image to generate the initial image.

[0083] (A5) In any of the embodiments indicated by any of (A1)-(A4), wherein the inverse dynamic range compression includes using DRC parameters.

[0084] (A6) In any of the embodiments indicated by any of (A1)-(A5), generating the low-frequency image and the high-frequency image from the initial image includes using decomposition parameters.

[0085] (A7) In any of the embodiments indicated by any of (A1)-(A6), the sparse interpolation includes sparse interpolation of the low-frequency image based on sub-sampled data.

[0086] (A8) In any of the embodiments indicated by any of (A1)-(A7), generating the low-frequency image and the high-frequency image further comprises decomposing the initial image using a low-pass filter according to the decomposition parameter.

[0087] (A9) In any of the embodiments indicated by any of (A1)-(A8), it also includes generating an original low-frequency image from the original image; and subsampling the original low-frequency image to generate subsampled image data. The sparse interpolation includes sparse interpolating the low-frequency image according to the subsampled image data.

[0088] (B1) A method for generating an image reconstruction metadata set, comprising: generating an original low-frequency image and an original high-frequency image from an original image or an image derived from the original image using decomposition parameters; subsampling the original low-frequency image to generate subsampled data; filtering a rendered image using the decomposition parameters to obtain a high-frequency image, wherein the rendered image is derived from the original image; and determining a reconstruction matrix, the product of which and the high-frequency image is equal to the original high-frequency image.

[0089] (B2) In the embodiment indicated by (B1), the method further includes performing a demosaicing process on the original image to obtain a rendered image.

[0090] (B3) In the embodiment indicated by (B1) or (B2), further comprising

[0091] Dynamic range compression (DRC) parameters are used to apply DRC to the original image to obtain an encoded original image, and the generating step includes generating an original low-frequency image and an original high-frequency image from the encoded original image.

[0092] (B4) In any of the embodiments indicated by any of (B1)-(B3), wherein applying the DRC to the original image includes gamma encoding the original image according to the DRC parameters.

[0093] (B5) In any of the embodiments indicated by any of (B1)-(B4), applying the DRC to the original image includes using a lookup table, and compressing the dynamic range of the original image includes using a lookup table.

[0094] (B6) In any of the embodiments indicated by any of (B1)-(B5), the method further includes determining the reconstruction matrix as a matrix that is multiplied by the high frequency image to produce the original high frequency image.

[0095] (C1) A method for reconstructing an image, comprising: generating sub-sampling data and a reconstruction matrix by generating an original low-frequency image and an original high-frequency image from an original image or an image derived therefrom using decomposition parameters, sub-sampling the original low-frequency image to generate the sub-sampling data, and filtering a rendered image using the decomposition parameters to produce a high-frequency image, wherein the rendered image is derived from the original image; and determining a reconstruction matrix, wherein the product of the reconstruction matrix and the high-frequency image is equal to the original high-frequency image; sparsely interpolating the low-frequency image using the sub-sampling data to generate a reconstructed low-frequency image; linearly estimating the original high-frequency image, which is produced by multiplying the high-frequency image by the reconstruction matrix; and generating a reconstructed original image from the reconstructed low-frequency image and the reconstructed high-frequency image.

[0096] (C2) In the embodiment indicated by (C1), it also includes using DRC parameters to perform inverse dynamic range compression on the reconstructed original image to generate a reconstructed original image.

[0097] (C3) In the embodiment indicated by (C1) or (C2), wherein the DRC parameters include encoding gamma, performing inverse dynamic range compression on the reconstructed original image further includes applying inverse gamma correction using the encoding gamma.

[0098] (C4) In any of the embodiments indicated by any of (C1)-(C3), the DRC parameters include a lookup table, and performing inverse dynamic range compression on the reconstructed original image also includes using the lookup table.

[0099] (D1) A system comprising: a processor; and a memory communicatively coupled to the processor and storing machine-readable instructions which, when executed by the processor, cause the processor to implement any one of methods (A1)-(A9), (B1)-(B6) and (C1)-(C4).

Claims

1. A method for generating an image reconstruction metadata set from an original image or an image derived from the original image and a rendered image derived from the original image, comprising: generating an original low-frequency image and an original high-frequency image from the original image or an image derived from the original image using the decomposition parameters; Sub-sampling the original low-frequency image to generate sub-sampled data; filtering the rendered image using the decomposition parameters to generate a high frequency image; as well as A reconstruction matrix is ​​determined, wherein a product of the reconstruction matrix and the high frequency image is equal to the original high frequency image.

2. The method according to claim 1, further comprising: Apply dynamic range compression DRC to the original image using the DRC parameters to obtain the encoded original image. The generating step includes generating an original low-frequency image and an original high-frequency image from the encoded original image. 3 . The method of claim 2 , wherein applying the DRC to the original image comprises gamma encoding the original image according to the DRC parameters.

4. The method of claim 2 or 3, wherein applying DRC to the original image comprises using a lookup table, and applying dynamic range compression to the original image comprises using a lookup table.

5. A method for reconstructing an original image from a rendered image derived from the original image, comprising: receiving subsampled data and a reconstruction matrix, wherein the subsampled data represents a subsampled original low-frequency image generated from the original image, and wherein the reconstruction matrix is ​​determined such that a product of the reconstruction matrix and a high-frequency image generated from the rendered image is equal to an original high-frequency image generated from the original image; sparsely interpolating a low-frequency image generated by the rendered image using the sub-sampled data to generate a reconstructed original low-frequency image; generating a reconstructed original high-frequency image as a product of a high-frequency image generated by the rendered image and a reconstruction matrix; and A reconstructed original image is generated from the reconstructed original low-frequency image and the reconstructed original high-frequency image. 6 . The method of claim 5 , wherein the low frequency image and the high frequency image are generated by decomposing the rendered image using a low pass filter according to a decomposition parameter. 7 . The method according to claim 5 , further comprising performing inverse dynamic range compression on the reconstructed original image using the DRC parameters to generate a further reconstructed original image.

8. A system comprising: processor; A memory, the memory being communicatively coupled to the processor and storing machine-readable instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images