Image processing method, device and storage medium

Abstract

The present application provides an image processing, device and storage medium. The method comprises the following steps: obtaining an initial image I; providing a set of m predetermined tone mapping functions; selecting a pixel I(i) of the initial image; determining a pixel weight w(i) of the selected pixel I(i) based on at least pixel information of the selected pixel; determining a tone mapping function weight for each tone mapping function based on the pixel weight; applying each tone mapping function to the selected pixel I(i) and weighting the result with the respective tone mapping function weight; and determining the sum of the weighted results as the final pixel I'(i).

Description

Technical Field

[0001] The invention relates to an image processing method, a device and a storage medium. Background Art

[0002] Photography is becoming more and more popular. At the same time, the requirements for image quality are also increasing. Among other things, the information captured by camera chips such as CMOS (Complementary Metal-Oxide-Semiconductor) is further processed in order to obtain the final image. An important step in this processing is tone mapping.

[0003] Tone mapping is the process of mapping image pixels that represent a relatively high dynamic range to a viewing environment (i.e., display media) with a relatively lower dynamic range. In doing so, the tone mapping process is responsible for providing an image that is presented as close to the real scene as possible. Tone mapping includes brightness and contrast enhancement sub-algorithms to produce subjectively pleasing images. The tone mapping algorithm can be located in a unique algorithm (for example, as a post-processing algorithm) or can be implemented as part of an image processing pipeline (also known as an Image Signal Processor (ISP) pipeline). The ISP can also be implemented in software, firmware, hardware, or a dedicated processor. The ISP can also be implemented in a simulation environment, which can be used to develop algorithms for end-user devices.

[0004] Common global and local tone mapping algorithms rely on decomposing an image into multiple layers and enhancing each layer independently before folding it back into the image. The Laplacian Pyramid-based Local Tone Mapping (TM) algorithm is one of such algorithms and is considered to be the state-of-the-art algorithm known from local Laplacian filters such as Edge-aware Image Processing with a Laplacian Pyramid by S. Paris, S. W. Hasinoff, J. Kautz, et al. These image-based multi-scale tone mapping algorithms are not favorable in the industry due to high line buffering costs (i.e., large hardware footprint) or high memory costs for offline processing. Therefore, in the industry, grid-based local tone mapping algorithms are used instead of decomposition-based multi-layer tone mapping algorithms. In it, the image is mapped onto a grid, wherein each node of the grid provides parameters for a specific region of the image. However, grid-based tone mapping algorithms suffer from the large number of parameters required by the algorithm. This increases the bandwidth cost, which is disadvantageous in real-time applications. Another disadvantage of grid-based solutions is that, depending on the interpolation algorithm used, these solutions may produce halos around high-contrast edges. These halos can be avoided if more advanced interpolation methods are used. However, advanced interpolation methods require additional line buffers, which increases the hardware implementation cost. Summary of the invention

[0005] It is therefore an object of the present invention to overcome the disadvantages of the prior art.It is further an object of the present invention to provide an improved tone mapping, thereby reducing hardware requirements and improving bandwidth.

[0006] According to the invention, this problem is solved by an image processing method according to claim 1 , a device according to claim 11 and a software according to claim 12 .

[0007] In one aspect of the present invention, there is provided an image processing method, in particular a tone mapping method, comprising the following steps:

[0008] a) obtaining an initial image I;

[0009] b) providing a set of m predetermined tone mapping functions;

[0010] c) Selecting a pixel I(i) of the initial image;

[0011] d) determining a pixel weight w(i) of the selected pixel I(i) based on at least the pixel information of the selected pixel;

[0012] e) determining a tone mapping function weight for each tone mapping function based on the pixel weight w(i);

[0013] f) applying each tone mapping function to the selected pixel I(i) and weighting the result with the respective tone mapping function weight;

[0014] g) The sum of the weighted results is determined as the final pixel I'(i).

[0015] In step a), an initial image I is preferably acquired from a camera device such as a mobile terminal by an image sensor. The initial image comprises pixels with pixel index i.

[0016] In step b), a set of m predetermined tone mapping functions is provided, wherein, preferably, m is equal to or greater than 2. The tone mapping function generally provides a mapping function that maps pixel information of a pixel with a high dynamic range to a lower dynamic range that can be displayed, for example, by a display device with a lower dynamic range. The tone mapping function can be implemented by hardware, software or firmware.

[0017] In step c), a pixel I(i) is selected from the initial image and according to step d), a pixel weight w(i) of the selected pixel I(i) is determined based on at least the pixel information of the selected pixel. Therein, the pixel information of the selected pixel can be used directly to determine the corresponding pixel weight. Alternatively, as described in more detail below, the pixel information is combined with further information, preferably of the initial image, in order to calculate the respective pixel weights and thus indirectly used to determine the pixel weights. According to step e), for each tone mapping function of the set of predetermined tone mapping functions, a tone mapping function weight is determined based on the pixel weight w(i) of the selected pixel I(i). In step f), each tone mapping function from the set of predetermined tone mapping functions is applied to the selected pixel I(i), wherein the respective results are weighted with the respective tone mapping function weights. Thus, m different tone mapping pixels are determined, each tone mapping pixel being weighted by the previously determined m tone mapping function weights. In a final step g), the sum of the weighted results is determined as the final pixel I'(i). Therefore, the weighted sum of the m tone-mapped original pixels weighted with the respective tone mapping function weights is used as the final pixel I'(i).

[0018] Therefore, according to the present invention, a set of a certain number of predetermined tone mapping functions is used. It is not necessary to implement or adjust a respective tone mapping function for each pixel or group of pixels individually. Therefore, the number of parameters required to apply tone mapping is reduced, and the required bandwidth is improved without losing image quality or reducing the appearance of the final image.

[0019] Preferably, steps c) to g) are repeated in particular for each pixel of the initial image in order to form the final image. Thus, a suitable tone mapping is provided for each pixel of the initial image.

[0020] Preferably, the tone mapping function is implemented as a look-up table (LUT), wherein brightness and / or contrast adjustment is provided by the LUT.

[0021] Preferably, the pixel weight w(i) is determined based on the pixel information of the selected pixel I(i). Wherein, the pixel information includes color information and / or brightness. In particular, if the Lab color space (also usually expressed as L*a*b* color space) is used, the pixel information can be related to one of the L channel, the a channel or the b channel or a combination thereof. More preferably, the pixel weight is determined based on the pixel information of the selected pixel and a plurality of adjacent pixels adjacent to the selected pixel or in the same area. Preferably, the adjacent pixels are arranged in a square around the selected pixel, etc. For example, the pixel weight is determined based on a weight grid, wherein a pixel weight is provided for each node of the weight grid, or a node value for calculating the pixel weight is assigned to each node. Each pixel in the initial image I is assigned to a respective grid node. Wherein, the weight grid is determined by interpolating the pixels assigned to the respective grid nodes or represented by the respective grid nodes, averaging the pixel information, filtering or any other method of approximating the pixel information of the pixels assigned to the respective nodes.

[0022] Preferably, the pixel weights are calculated by interpolation based on the pixel position provided by the pixel index i and the values ​​of the neighboring nodes, i.e., the respective node values, and / or the pixel information of the neighboring pixels. This interpolation can be performed in various ways, for example, using bilinear, bicubic or more advanced interpolation algorithms, such as bilateral interpolation algorithms. Depending on the interpolation method adopted, additional information can be used to support the pixel weight grid interpolation. For example, for joint bilateral interpolation, a reduced (i.e., low-resolution) representation of the input image can be used for better guidance during interpolation.

[0023] Preferably, the pixel weight is based on brightness as pixel information.

[0024] Preferably, each tone mapping function is associated with an indication value providing information on the association between the tone mapping function and the respective pixel weight.

[0025] Preferably, the indication value is a brightness value.

[0026] Preferably, the indicator values ​​are distributed between a minimum indicator value and a maximum indicator value, wherein the minimum indicator value may be selected according to one of the following:

[0027] -Minimum brightness value of the initial image;

[0028] - an absolute brightness minimum, usually represented by the value 0; or

[0029] - Preset minimum brightness value set by the user.

[0030] Alternatively or additionally, the maximum indication value may be selected according to one of the following:

[0031] -The maximum brightness value of the initial image;

[0032] - the absolute maximum brightness value, usually represented by the value 1;

[0033] - A preset maximum brightness value set by the user before or during the execution of the steps of the method.

[0034] Therein, all of the above-mentioned options for the minimum indication value can be freely combined with all of the options for the maximum indication value.

[0035] Preferably, the tone mapping functions associated with the respective indicator values ​​are evenly distributed between the minimum indicator value and the maximum indicator value.

[0036] Preferably, the tone mapping function weights are calculated based on the difference between the indicator value and the pixel weight.

[0037] Preferably, the tone mapping function weight w is calculated according to the following formula: k , k=1,...,m:

[0038]

[0039] Among them, w(i) represents the pixel weight, and a k Indicates the indicator value of the respective tone mapping function.

[0040] Preferably, in order to ensure that the tone mapping function weights are between [0, 1], the tone mapping function weights are normalized, in particular by the following formula:

[0041]

[0042] In another aspect of the present invention, a device comprising an image sensor and a processor is provided, wherein the processor is configured to perform the steps of the aforementioned method. In particular, the processor is connected to a memory storing a predetermined tone mapping function.

[0043] In another aspect, a storage medium is provided, wherein the storage medium stores executable instructions, and when the executable instructions are executed by a computing device, the computing device executes the steps of the aforementioned method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The present invention is further described with reference to the accompanying drawings.

[0045] The accompanying drawings show:

[0046] Figure 1 is a flow chart of a method according to the present invention;

[0047] Figure 2A is an example of a weight grid;

[0048] Figure 2B is an example of a set of predetermined tone mapping functions;

[0049] Figure 3 is a diagram showing the calculation of the tone mapping function weight according to the present invention;

[0050] Figure 4 A diagram for calculating the final pixel;

[0051] Figure 5 For comparison of the present invention;

[0052] Figure 6 is the device according to the present invention. DETAILED DESCRIPTION

[0053] Reference Figure 1 , shows a flow chart of an image processing, in particular a tone mapping method.

[0054] In step S01 , an initial image I is acquired, for example, by a camera device including a camera or an image sensor of a mobile terminal.

[0055] In step S02, a set of m predetermined tone mapping functions is provided. The number of predetermined tone mapping functions is equal to or greater than 2. In particular, m is selected to be equal to or greater than 2 and less than or equal to 20 (2≤m≤20), and more preferably, less than or equal to 10 (2≤m≤10). More preferably, the number of predetermined tone mapping functions is equal to or greater than 2 and equal to or less than the number of nodes of the weight grid, as described in more detail below.

[0056] In step S03, a pixel I(i) of the initial image is selected.

[0057] In step S04, the pixel weight w(i) of the selected pixel I(i) is determined based on the pixel information of at least the selected pixel. Wherein, the pixel information may include color information and / or brightness. For example, when using the Lab color space, the pixel information may include an L channel, an a channel or a b channel or a combination thereof. More preferably, the pixel weight w(i) is determined based on brightness, as described in more detail below. Wherein, preferably, the pixel information of the selected pixel is used to calculate a weight grid, wherein the weight grid is used to calculate the pixel weight w(i), as described below.

[0058] In step S05 , for each tone mapping function, a tone mapping function weight is determined based on the pixel weight.

[0059] In step S06, each tone mapping function is applied to the selected pixel I(i) and the result is weighted by the respective tone mapping function weight. Thus, firstly a tone mapping pixel is determined for each tone mapping function in a set of predetermined tone mapping functions. Where k = 1, ..., m. Therefore, m results of the selected pixel of the tone mapping are obtained. Then, by Multiplied by their respective tone mapping function weights w k (i) Weight each tone-mapped pixel To obtain the weighted tone-mapped pixels

[0060] In step S07, the sum of the weighted results Determined as the final pixel I'(i).

[0061] Therefore, a set of a certain number m of predetermined tone mapping functions is used. It is not necessary to implement or adjust a respective tone mapping function for each pixel or group of pixels individually. Therefore, the number of parameters required to apply tone mapping is reduced, and the required bandwidth is improved without losing image quality or degrading the appearance of the final image.

[0062] Reference Figure 2A , shows a weight grid 10 having a plurality of grid nodes 12, wherein for each pixel I(i), a weight may be assigned according to its position in the weight grid 10. For example, Figure 2A The illustrated pixel 14 is assigned to the upper left grid node 12. Thus, the pixel weight of the exemplary pixel 14 may be provided by the grid node 12 or may be calculated based on the node value assigned to the corresponding grid node 12.

[0063] Wherein, the weight grid 10 can be calculated by interpolation based on the pixel information in the area covered by the respective weight grid nodes 12 or the pixel information of the pixels assigned to the respective grid nodes 12. Interpolation can be performed by averaging the pixel information and using the average value as the pixel weight of all pixels assigned to the corresponding grid nodes 12. Wherein, the pixel information involves color channel information and / or brightness. For example, when using the Lab color space, the weight grid 10 is determined based on one or more of the L channel, the a channel and / or the b channel. Other possibilities other than averaging the pixel information to calculate the weight grid 10 can be implemented, and may include, for example, machine learning algorithms, etc. However, the present invention is not limited to a specific way of determining the weight grid 10. The determination (construction, creation) of the weight grid 10 can be performed in hardware and software, that is, implemented in software as a learning-based algorithm.

[0064] In addition, pixel weights can be obtained by interpolation based on the values ​​assigned to neighboring nodes and / or pixel information of neighboring pixels of the selected pixel. Among them, this interpolation can be performed in various ways, for example, using bilinear, bicubic or more advanced interpolation algorithms, such as bilateral interpolation algorithms. Depending on the interpolation method adopted, additional information can be used to support per-pixel weight grid interpolation. For example, for bilateral interpolation, a reduced (i.e., low-resolution) representation of the input image can be used for better guidance during interpolation.

[0065] Furthermore, the number of grid nodes 12 can be selected and in the present example is selected as a grid with 12×16 grid nodes 12. The selection of the number of nodes 12 can be performed based on the initial image, for example in relation to contrast variations in the image, by fixing the current number for several initial images, or can be set by a user. Therein, for a large number of nodes 12, the method applies local adaptation of the respective regions of the initial image, while for a small number of nodes 12, global adaptation is applied. A grid node value preferably between [0, 1] can be assigned to each grid node 12. Thus, when selecting a specific pixel I(i) of the initial image I indicated by the pixel index i, the pixel weight w(i) can be determined based on the weight grid 10, wherein the weight grid 10 provides weights for the selected pixel based on at least pixel information of the selected pixel and preferably by pixels belonging to the same grid node 12.

[0066] Reference Figure 2B , a set of m predetermined tone mapping functions is provided, preferably implemented as a lookup table (LUT). Figure 2B , the respective tone mapping functions are from LUT1 to LUT m The number of predetermined tone mapping functions is preset and is at least greater than 2. More preferably, the number of predetermined tone mapping functions is between 2 and 20, more preferably between 2 and 10. Most preferably, the number of preset tone mapping functions is greater than 2 and less than or equal to the number of nodes 12 of the weight grid 10. Each tone mapping function is associated with an indication value a1 to a m The indicator value provides information about the association between the respective tone mapping function and the respective weight value of the weight grid 10. If the values ​​of the weight grid 10 are restricted to [0, 1], the indicator value is also restricted to these values. In particular, the indicator value is related to the brightness value. The indicator value may be distributed between a minimum indicator value and a maximum indicator value, wherein the minimum indicator value and / or the maximum indicator value are selected according to one of the following:

[0067] - the minimum or maximum brightness value of the initial image;

[0068] - an absolute minimum brightness value, usually represented by a value of 0, or an absolute maximum brightness value, usually represented by a value of 1; or

[0069] - Preset minimum brightness value or preset maximum brightness value.

[0070] Reference Figure 3 According to step S05, for each tone mapping function, determine the respective tone mapping function weights w1(i) to w m (i). Starting from the weight grid 10, the pixel weight w(i) of the selected pixel I(i) is calculated. Then, based on the pixel weight w(i), a separate tone mapping function weight is calculated for each tone mapping function. The algorithm starts by calculating the weight for a given pixel index i, for example, by interpolating the relevant weight grid nodes 12. This interpolation can be performed in various ways, for example, using bilinear, bicubic or more advanced interpolation algorithms, such as bilateral interpolation algorithms.

[0071] For a given pixel index with calculated pixel weight w(i) and node information, calculate the tone mapping function weights for each LUT, i.e., w1(i),…,w n (i), …, w m (i). The indicator value carries the association of each LUT with the interpolation weight for the image index i. Each tone mapping function weight can then be calculated using any interpolation algorithm that uses any distance information, such as absolute distance, squared distance, or any combination thereof. Interpolation can also be performed between the selected nearest (2 or a few) nodes 12. For example, for a LUT k and image index i, the weight w can be calculated by using the squared distance based on the following formula k (i)

[0072]

[0073] Among them, a k is the indicated value for each LUT.

[0074] In order to ensure that w1(i),…,w m (i), …, w n (i) Between [0, 1], the weights can be further normalized by:

[0075]

[0076] The calculated tone mapping function weights are then passed to the next step and used for fusion of the output of the LUT.

[0077] Reference Figure 4, shows the application of the LUTs and the calculation of the respective tone-mapped pixels. In this step, each LUT is first applied to the selected image pixel I(i) to determine the output Then, by using the previously calculated tone mapping function weights w1(i),…,w n (i), …, w m (i), with their respective tone mapping function weights w1(i),…,w n (i), …, w m (i) Weighted selected pixels of all tone maps The weighted sum of the pixels on the final tone map determines the final tone map pixel I 色调映射 (i) (or I'(i)). In other words,

[0078]

[0079] These steps are repeated for each pixel of the initial image to form a correctly tone-mapped final image.

[0080] The present invention provides an effective and efficient local tone mapping method (both brightness and contrast are enhanced). Because the present invention does not require any line buffers, it can be efficiently implemented in hardware. The number of parameter sets is not as many as other grid-based tone mapping algorithms commonly used in the industry. Therefore, the required bandwidth is smaller. In addition, the method is fully scalable between local and global operations. When the number m of predetermined tone mapping functions decreases and / or the provided LUTs become more similar (similar to each other), the algorithm starts to operate more in a global manner. When the LUTs are different from each other and / or the number m increases, the process becomes more locally oriented.

[0081] Reference Figure 5 , provides examples generated by the present invention. Especially in high-dynamic range (HDR) scenes, in addition to global operations, local tone mapping and contrast optimization operations are crucial to generating pleasing images. Figure 5 In Figure 1, the HDR image was processed as follows: no tone mapping (left image), only global LUT (middle image), and global and local operations (performed by the method of the present invention) (right image). As can be seen from the rightmost column, the local operations are particularly responsible for bringing out the local details (see the contrast of clouds, trees and buildings in the sky). These local details cannot be enhanced using global operations.

[0082] Reference Figure 6, a device such as a terminal device 200 including a camera sensor or image sensor 220 connected to a processor 240 is shown. In addition, the processor 240 is connected to a memory 260. The memory 260 may store instructions to be executed by the processor 240 to transform the raw image data captured by the image sensor 220 and provide the final image to the user of the terminal device 200. Among them, the processor 240 and the memory 260 may be implemented as a dedicated image processor, the ISP may be implemented in hardware, or the ISP may be implemented in software stored in the memory 260 and processed by the processor 240.

Claims

1. An image processing method, comprising: a) obtaining an initial image I; b) providing a set of m predetermined tone mapping functions; c) Select pixel I(i) of the initial image; d) determining a pixel weight w(i) of the selected pixel I(i) based on pixel information of the selected pixel, wherein the pixel information includes color information and / or brightness; e) determining a tone mapping function weight for each tone mapping function based on the pixel weight; f) applying each tone mapping function to the selected pixel I(i) and weighting the result with the respective tone mapping function weight; g) determining the sum of the weighted results as the final pixel I'(i); wherein each tone mapping function is associated with an indicator value, wherein the indicator value is a brightness value; The tone mapping function weight is calculated based on the difference between the indication value and the pixel weight.

2. The method according to claim 1, wherein: Repeat steps c) to g) for each pixel of the initial image.

3. The method according to claim 1 or 2, wherein: The tone mapping function is implemented as a look-up table LUT.

4. The method according to any one of claims 1 to 3, wherein: The pixel weight is determined based on pixel information of the selected pixel and a plurality of neighboring pixels adjacent to the selected pixel.

5. The method according to claim 1, wherein: The indication value is distributed between a minimum indication value and a maximum indication value, wherein the minimum indication value and / or the maximum indication value is selected according to one of the following: - the minimum brightness value or the maximum brightness value of the initial image; - the absolute minimum luminance value or the absolute maximum luminance value; or - Preset minimum brightness value or preset maximum brightness value.

6. The method according to any one of claims 1 to 5, wherein: The tone mapping function weight is calculated according to the following formula , k=1,...,m: Wherein, w(i) represents the pixel weight, Indicates the indicated value.

7. The method according to claim 6, wherein: The tone mapping function weights are normalized by: 。 8. A device comprising an image sensor and a processor, wherein: The processor is configured to perform the steps of the method according to any one of claims 1 to 7.

9. A storage medium storing executable instructions, wherein: When the executable instructions are executed by a computing device, the computing device is caused to perform the steps of the method according to any one of claims 1 to 7.

Images