Image processing method, device and storage medium

By mapping the image to a locally unchanged space, using monotonic incremental transfer function and preset tone mapping function, the problems of large hardware occupation and high bandwidth in the prior art are solved, and efficient tone mapping and contrast enhancement effects are achieved.

CN115147291BActive Publication Date: 2024-11-08BEIJING XIAOMI MOBILE SOFTWARE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210225422.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-03-30
Filing Date
2022-03-09
Publication Date
2024-11-08
Estimated Expiration
2042-03-09

AI Technical Summary

Technical Problem

Existing grid-based local tone mapping algorithms take up a lot of space in hardware implementations, have high bandwidth costs, and may create halos at high contrast edges, and advanced interpolation methods increase hardware costs.

Method used

By mapping the initial image to a locally unchanged image space, using monotonic incremental and reversible transfer functions and preset tone mapping functions, the number of necessary parameters is reduced, and tone mapping is realized, avoiding adapting to different tone mapping functions for each pixel. demand.

Benefits of technology

Reduces hardware requirements and bandwidth costs, improves the performance of the tone mapping process, and avoids halo problems at high contrast edges, achieving efficient image processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115147291B_ABST
    Figure CN115147291B_ABST
Patent Text Reader

Abstract

The present application discloses an image processing method, device and storage medium, wherein the method comprises the following steps: obtaining an initial image I; selecting a pixel I(i) of the initial image; applying a transfer function f to the pixel I(i) to obtain a transfer pixel #imgabs0#, wherein the transfer function f is determined based on at least pixel information of the selected pixel; applying a predetermined tone mapping function to the transfer pixel #imgabs1# to obtain a tone-mapped transfer pixel #imgabs2#; applying an inverse function f of the transfer function to the transfer pixel #imgabs1#; ‑1 The tone mapped transfer pixel #imgabs3# is applied to obtain the tone mapped final pixel I′(i). These steps are repeated for multiple pixels of the initial image I to generate the final image I′, wherein the same tone mapping function is used for each of the multiple pixels.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an image processing method, a device and a storage medium. Background Art

[0002] Photography is becoming more and more popular. At the same time, the requirements for image quality are also increasing. Among other things, the information captured with the camera chip (e.g., CMOS, etc.) is further processed in order to obtain the final image. An important step in this processing is tone mapping.

[0003] Photography is becoming more and more popular. At the same time, the requirements for image quality are also increasing. Among other things, the information captured by camera chips such as CMOS (Complementary Metal-Oxide-Semiconductor) is further processed in order to obtain the final image. An important step in this processing is tone mapping.

[0004] Tone mapping is the process of mapping image pixels that represent a relatively high dynamic range to a viewing environment (i.e., display media) with a relatively lower dynamic range. In doing so, the tone mapping process is responsible for providing an image that is presented as close to the real scene as possible. Tone mapping includes brightness and contrast enhancement sub-algorithms to produce subjectively pleasing images. The tone mapping algorithm can be located in a unique algorithm (for example, as a post-processing algorithm) or can be implemented as part of an image processing pipeline (also known as an Image Signal Processor (ISP) pipeline). The ISP can also be implemented in software, firmware, hardware, or a dedicated processor. The ISP can also be implemented in a simulation environment, which can be used to develop algorithms for end-user devices.

[0005] Common global and local tone mapping algorithms rely on decomposing an image into multiple layers and enhancing each layer independently before folding it back into the image. One such algorithm is the Laplacian Pyramid-based Local Tone Mapping (TM) algorithm, which is considered to be the state-of-the-art algorithm known from local Laplacian filters such as Edge-aware Image Processing with a Laplacian Pyramid by S. Paris, S. W. Hasinoff, J. Kautz, et al. These image-based multi-scale tone mapping algorithms are not favorable in the industry due to the high line buffering cost (i.e., large hardware footprint) for offline processing. Therefore, in the industry, grid-based local tone mapping algorithms are used instead of decomposition-based multi-layer tone mapping algorithms. In this, the image is mapped onto a grid, wherein each node of the grid provides parameters for a specific region of the image. However, grid-based tone mapping algorithms suffer from the large number of parameters required by the algorithm. This increases the bandwidth cost, which is disadvantageous in real-time applications. Another disadvantage of grid-based solutions is that, depending on the interpolation algorithm used, these solutions may produce halos around high-contrast edges. These halos can be avoided if more advanced interpolation methods are used. However, advanced interpolation methods require additional line buffers, which increases the hardware implementation cost. Summary of the invention

[0006] Therefore, it is an object of the present invention to overcome the drawbacks of the prior art.Further, it is an object of the present invention to provide an improved tone mapping, thereby reducing hardware requirements and improving bandwidth.

[0007] According to the invention, this problem is solved by an image processing method according to claim 1 , a device according to claim 10 and software according to claim 11 .

[0008] In one aspect of the present invention, there is provided an image processing method, in particular a tone mapping method, comprising the following steps:

[0009] a) obtaining an initial image I;

[0010] b) Selecting a pixel I(i) of the initial image;

[0011] c) Apply the transfer function f to pixel I(i) to obtain the transfer pixel wherein the transfer function f is determined based on pixel information of at least the selected pixel;

[0012] d) Apply a predetermined tone mapping function to the transfer pixels To get the tone mapped pass pixels

[0013] e) The inverse function of the transfer function f -1 The passed pixels to be applied to the tone map To obtain the final pixel I'(i) of the tone mapping;

[0014] Steps b) to e) are repeated for a plurality of pixels of the initial image I to generate a final image I', wherein the same tone mapping function is used for each of the plurality of pixels.

[0015] In step a), preferably, an initial image is acquired by an image sensor of an image device such as a mobile terminal, etc. In step b), a pixel I(i) is selected from the initial image, where i is a pixel index.

[0016] In step c), a transfer function f is applied to the pixel I(i), wherein the transfer function f is determined based on the pixel information of at least the selected pixel I(i). The pixel I(i) is mapped or transferred to a locally invariant image space by applying the transfer function f so as to obtain a transferred pixel Wherein, the transfer function is preferably monotonically increasing and reversible. Preferably, for the entire initial image, the function type (e.g., gamma function, exponential function, polynomial function of a certain degree, etc.) is preset and fixed, and only one or more function parameters of the function, i.e., function constants, are adjusted. Wherein, the pixel information of the selected pixel can be used directly to determine the function or function parameters. Alternatively, as described in more detail below, the pixel information is combined with further information, preferably with information of the initial image, in order to calculate the respective function or function parameters, thereby indirectly used to determine the function or function parameters.

[0017] In step d), a predetermined tone mapping function is applied to the transfer pixels In order to obtain tone-mapped transferred pixels in a locally invariant image space

[0018] In step e), the inverse function f of the transfer function is -1 The passed pixels to be applied to the tone map In order to transfer the tone-mapped pixels Return to the original image space and obtain the final or tone-mapped pixel I(i).

[0019] Therein, steps b) to e) are repeated for a plurality of pixels of the initial image I in order to generate a final image I', wherein for each of the plurality of pixels the same tone mapping function is used. Therefore, it is no longer necessary to adapt the respective tone mapping function to the respective pixel I(i) of the initial image. In the present invention, the opposite is the case, since the initial image is mapped into a locally invariant image space by means of a transfer function, wherein the transfer function is adapted to the respective pixel. Therefore, it is no longer necessary to provide or adjust a large number of tone mapping functions. Instead, the transfer function f is adjusted accordingly. Therefore, the number of necessary parameters can be reduced, thereby improving the bandwidth and the performance of the tone mapping process.

[0020] Preferably, the same predetermined tone mapping function is used for all pixels of the initial image in order to form the final image I'. Hence, the parameters of only one tone mapping function have to be provided to the current tone mapping process.

[0021] Preferably, the tone mapping function is provided by a look-up table (LUT) or a so called s-curve.

[0022] Preferably, a function f is determined so as to map the selected pixel to a value for each pixel of the initial image, wherein the value is determined based on a local mean and preferably such that the local mean becomes 0.5 (for the example of a Lab color space ranging between [0, 1]) and / or such that the local mean coincides with an inflection point of the tone mapping function.

[0023] Preferably, the tone mapping function implements contrast manipulation and / or brightness manipulation in order to provide a pleasing result for the final image I'.

[0024] Preferably, the transfer function, and more preferably one or more function parameters, is determined based on a plurality of pixels immediately adjacent to the selected pixel I(i). In particular, for determining the transfer function f, pixels arranged in a square around the selected pixel I(i) are considered.

[0025] Preferably, the transfer function, and more preferably one or more function parameters, are determined based on a local adaptation parameter (LAP) grid, wherein each pixel of the initial image is assigned to one of the grid nodes of the LAP grid, wherein each grid node indicates a respective local adaptation parameter. The respective local adaptation parameter of each grid node is determined based on a plurality and preferably all pixels assigned to the respective grid node. Hence, the local adaptation parameter of the grid node provides an estimate of the pixel assigned to the respective grid node.

[0026] Preferably, the grid is determined directly after step a) of acquiring the initial image I.

[0027] Preferably, the transfer function, and more preferably one or more function parameters, are determined by interpolation based on local adaptation parameters of neighboring nodes and / or pixel information of pixels of the initial image.

[0028] Preferably, the pixel information may refer to one or more color channels and / or brightness of the respective color space. For the example of the Lab color space (also referred to as the L*a*b* color space), the pixel information used to determine the grid value of the grid may be the a channel, the b channel, the L channel, or a combination of more than one of these channels.

[0029] Preferably, the transfer function is determined by the brightness of the pixel as pixel information.For the example of Lab color space, the L channel is used to determine the transfer function f or the LAP grid.

[0030] Preferably, the transfer function f is implemented as a gamma function.

[0031] In another aspect of the present invention, a device comprising an image sensor and a processor is provided, wherein the processor is configured to perform the steps of the aforementioned method. In particular, the processor is connected to a memory storing a predetermined tone mapping function.

[0032] In another aspect, a storage medium is provided, which stores executable instructions. When the executable instructions are executed by a computing device, the computing device is caused to perform the aforementioned method steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Hereinafter, the present invention will be further described with reference to the accompanying drawings.

[0034] The accompanying drawings show:

[0035] Figure 1 is a flow chart of a method according to the present invention;

[0036] Figure 2 is a block diagram of a method according to the present invention;

[0037] Figure 3A Embodiments of the present invention for determining respective transfer functions;

[0038] Figure 3B Example of a locally adaptive parameter grid;

[0039] Figure 4 For comparison of the present invention;

[0040] Figure 5 is the device according to the present invention. DETAILED DESCRIPTION

[0041] Reference Figure 1, shows a flow chart of an image processing, in particular a tone mapping method.

[0042] In step S01 , an initial image I is acquired, for example, by a camera device including a camera or an image sensor of a mobile terminal.

[0043] In step S02, a pixel I(i) of the initial image is selected.

[0044] In step S03, the transfer function f is applied to the pixel I(i) to obtain the transfer pixel The selected pixel I(i) is transferred to the locally invariant image space by means of a transfer function f. The transfer function f is determined based on at least the pixel information of the selected pixel I(i). The transfer function f is preferably monotonically increasing and reversible. Preferably, for the entire initial image, the function type (e.g., a gamma function, an exponential function, a polynomial function of a certain degree, etc.) is preset and fixed, and only one or more function parameters of the function, i.e., the function constant, are adjusted.

[0045] Wherein, pixel information can refer to one or more color channels and / or brightness of respective color spaces. For the example of Lab color space (also referred to as L*a*b* color space), pixel information for determining the grid value of a grid can be a channel, b channel, L channel, or a combination of more than one of these channels. Wherein, L channel is also referred to as brightness, lightness, or brightness.

[0046] In step S04, a predetermined tone mapping function is applied to the transfer pixel To get the tone mapped pass pixels

[0047] In step S05, the inverse function of the transfer function f -1 The passed pixels to be applied to the tone map to obtain the final pixel I'(i) of the tone mapping back to the original image space.

[0048] Therein, according to the present invention, steps S02 to S05 are repeated for a plurality of pixels of an initial image I in order to generate a final image I', wherein the same tone mapping function is used for each of the plurality of pixels. Therefore, it is no longer necessary to adapt the respective tone mapping function to the corresponding pixel I(i) of the initial image. Instead, in the present invention the opposite is the case, since the initial image is mapped into a locally invariant image space by means of a transfer function, wherein the transfer function is applicable to the corresponding pixel. Therefore, it is no longer necessary to provide or adjust a large number of tone mapping functions, since the transfer function f is adjusted accordingly, and the number of necessary parameters can be reduced, thereby improving the bandwidth and performance of the tone mapping process.

[0049] Reference Figure 2First, for pixel index i, obtain the local adaptation parameter (LAP). Figure 3A , to determine the LAP of a given selected pixel 12 (denoted by LAP(i)), the pixels in the area surrounding the selected pixel 12 are considered. Figure 3A In the example, pixels in a 3×3 square 16 are considered. However, the present invention is not limited to a specific number of pixels considered for determining the LAP, nor to a specific arrangement of pixels considered as a square surrounding the selected pixel 12. When applying the interpolation algorithm, the LAP of the selected pixel 12 is calculated. Therein, the interpolation algorithm may average the pixel information of the pixels surrounded by the square 16, which is considered for determining the LAP(i) of the selected pixel 12, etc. The pixel information may include one or more color channels and / or brightness in the respective color space. More preferably, the LAP is based only on brightness. Therefore, for the example of the Lab color space, only the L channel is considered to determine the LAP of the selected pixel 12.

[0050] Figure 3B The local adaptive parameter grid 18 of the initial image is shown, wherein each pixel of the initial image is assigned to a specific grid node 20. Each grid node 20 has a LAP grid value calculated based on the pixel information of the assigned pixel of the respective grid node 20. Figure 3B In the example of , the initial image is represented by a 16×12 LAP grid 18. However, the number of grid nodes in the grid can be freely selected and is not limited to Figure 3B Example. Based on an initial image, for example an initial image related to contrast changes in the image, the selection of the number of nodes can be performed by fixing the current number for several initial images or can be set by the user. Therein, for a large number of nodes, the method applies local adaptation of the respective area of ​​the initial image, and for a small number of nodes, a global adaptation is applied. The determination (construction, creation) of the LAP grid can be performed both in hardware and in software, i.e. in software as a learning-based algorithm.

[0051] The determination of the LAP grid values ​​may depend on the transfer function f. If the LAP grid is to be calculated in software, the algorithm will need additional grid information, such as the local mean (or average) which provides the average value for each pixel assigned to the respective node. This grid may be denoted MG (MeanGrid). This grid may be calculated in hardware and passed to the software. Assuming that the function f is a gamma operation, each LAP grid value should map the respective MG value to the value 0.5 (which is the inflection point of the contrast manipulation curve). If the inflection point of a given LUT changes, the LAP grid value should also change. Therefore, for the LAP of grid node j, LAP(j) is:

[0052] LAP(j)=log(0.5) / log(MG(j)).

[0053] In another embodiment, instead of the gamma operation as a function, only the gain operation is used (ie, only the pixel is multiplied by a given value), then LAP(j) will be calculated by:

[0054] LAP(j)=0.5 / MG(j).

[0055] This means that applying the function f pulls the local mean towards 0.5.

[0056] In another embodiment, the LAP of a given index i can be calculated using any image interpolation algorithm to interpolate the LAP based on the pixel position given by the pixel index i, the neighboring nodes of the LAP grid and / or the pixel information of the neighboring pixels. For example, for this interpolation, simple algorithms such as bilinear, bicubic interpolation methods and more advanced interpolation algorithms such as bilateral or intensity distance-based interpolation algorithms can be used. Depending on the interpolation strategy adopted, the LAP grid may be accompanied by additional information. For example, in order to implement guided interpolation based on image intensity, a low-resolution local average grid can be used.

[0057] Return to reference Figure 2, the LAP of the selected pixel I(i) is used to determine the transfer function f, and preferably to determine the function parameters of f, so as to transfer the image pixel I(i) into the locally invariant image space and map the pixel I(i) to the inflection point of the tone mapping function. In order to achieve the desired contrast enhancement effect, the interpolated LAP(i) must be used to map the input image pixel I(i). The forward mapping of the input image pixel is performed by the function f(p,I). The forward mapping function is a complement to the LAP grid, and as mentioned above, the designs of the two are consistent. There are many ways to implement this function. For example, it can be as general as a lookup table (LUT) operator, or it can be used as a dedicated function like a gamma operator. Therefore, through the transfer function, the pixel I(i) of the initial image I is mapped to the transfer pixel in the locally invariant image space Afterwards, a predetermined tone mapping function is applied. The tone mapping function may be implemented as a lookup table (LUT). After the selected image pixels are mapped to the locally invariant image space, a contrast / brightness manipulation LUT is applied to obtain the tone mapped pixels in the locally invariant image space. This process can be well implemented as a typical LUT operation or it can also be implemented as a parameter or dedicated function as part of the hardware. Thus, by applying the LUT, tone-mapped pixels are created in the locally invariant image space. After the image pixels have been subjected to contrast and / or brightness manipulation in the locally invariant image space, they need to be remapped back to the original image space. This is done by the inverse mapping function f -1 (p,I) implements this inverse remapping, which also uses the interpolated local adaptive parameters LAP(i) of the selected pixel I(i). Therefore, through the inverse transfer function f -1 , the tone-mapped pixels is mapped back to the original color space to form the final tone-mapped pixel I'(i).

[0058] The present invention provides effective and efficient local contrast enhancement. Because the present invention does not require any line buffers, it can be efficiently implemented by hardware. The number of necessary parameters is not as many as other grid-based tone mapping algorithms commonly used in the industry. Therefore, the required bandwidth is smaller. In addition, for a given runtime and scene content adjustment requirements, the method is fully scalable between local and global contrast enhancement operations. When the local adaptive parameters in the LAP grid are set closer to each other, that is, when the number of nodes within the LAP grid is reduced, the algorithm begins to operate more in a global manner. When these parameters are different from each other, the number of nodes in the LAP grid is increased, and the process becomes more locally oriented. The scalability of the ISP algorithm is critical, for example, when the algorithm needs to run at different frame rates, different content, etc.

[0059] Below is an example of the present invention. In a particularly high-dynamic range (HDR) scene, a global contrast enhancement LUT cannot be used to obtain details in shadow and bright areas. The reason behind this is that in global operations, the processing of tones is independent of local areas. For example, mid-tones in highlights (relatively dark) and mid-tones in shadows (those mid-tones are relatively bright) are not distinguished from each other but are processed the same. Therefore, local contrast enhancement operations are needed to obtain local details to produce a pleasing image. Figure 4 In FIG. 5 , the following processed HDR images are shown: without tone mapping (left image), without local adaptation of any contrast enhancement (middle image), and with local contrast adaptation (performed by the method according to the invention) (right image). As can be seen from the rightmost column, the local operation is particularly responsible for obtaining local details (see the contrast of clouds, trees and buildings in the sky). Without local adaptation for contrast enhancement, these local details can only be enhanced to a certain level.

[0060] Reference Figure 5 , shows a device including a terminal device 200 such as a camera sensor or image sensor 220 connected to a processor 240. In addition, the processor 240 is connected to a memory 260. The memory 260 can store instructions to be executed by the processor 240 to transform the raw image data captured by the image sensor 220 and provide the final image to the user of the terminal device 200. Among them, the processor 240 and the memory 260 can be implemented as a dedicated image processor, the ISP can be implemented in hardware, or the ISP can be implemented in software stored in the memory 260 and processed by the processor 240.

Claims

1. An image processing method, comprising: a) obtaining (S01) an initial image I; b) selecting (S02) a pixel I(i) of the initial image; c) Apply the transfer function f to (S03) pixel I(i) to obtain the transfer pixel wherein the transfer function f is determined based on the pixel information of the selected pixel, and the pixel I(i) is mapped to a locally invariant image space based on the transfer function f; d) applying a predetermined tone mapping function to (S04) the transfer pixel To get the tone mapped pass pixels e) The inverse function of the transfer function f -1 Applying (S05) the tone-mapped transfer pixels To obtain the final pixel I'(i) of the tone mapping; Steps b) to e) are repeated for a plurality of pixels of the initial image I to generate a final image I', wherein the same tone mapping function is used for each of the plurality of pixels.

2. The method according to claim 1, wherein: The same predetermined tone mapping function is used for all pixels of the initial image.

3. The method according to claim 1 or 2, wherein: The tone mapping function is provided by a look-up table LUT.

4. The method according to any one of claims 1 to 3, wherein: The tone mapping function implements contrast manipulation and / or brightness manipulation.

5. The method according to any one of claims 1 to 4, wherein: The transfer function f is determined based on a plurality of pixels arranged in a square around the selected pixel I(i), next to the selected pixel.

6. The method according to any one of claims 1 to 5, wherein: The transfer function f is determined based on local adaptive parameters of the grid, wherein each pixel is assigned to a grid node and the respective local adaptive parameter of each node is determined from all pixels assigned to the respective grid node.

7. The method according to claim 6, wherein: The transfer function f is determined by interpolation based on local adaptation parameters of neighboring nodes and / or pixel information of the plurality of pixels of the initial image.

8. The method according to claim 6 or 7, wherein: The transfer function f is determined based on the brightness of the plurality of pixels.

9. The method according to any one of claims 1 to 8, wherein: The transfer function f is implemented as a gamma function.

10. A device comprising an image sensor and a processor, wherein: The processor is configured to perform the steps of the method according to any one of claims 1 to 9.

11. A storage medium storing executable instructions, wherein: The executable instructions, when executed by a computing device, cause the computing device to perform the steps of the method according to any one of claims 1 to 9.