Methods and apparatus for scene-dependent lens shading correction

By collecting scene information from the Bayer thumbnails of the image, simulating white balance and pre-gamma blocks, generating standard red-green-blue thumbnails, determining representative color channel ratios and merging gains, the problem of inconsistent color rendering caused by lens shading is solved, improving image quality and user satisfaction.

CN115334261BActive Publication Date: 2026-11-17SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210511147.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2022-01-10
Filing Date
2022-05-11
Publication Date
2026-11-17
Estimated Expiration
2042-05-11

AI Technical Summary

Technical Problem

Existing lens shadow correction techniques cannot effectively eliminate color artifacts in images, especially when shooting under different lighting conditions, where lens shadows cause inconsistent color rendering.

Method used

By collecting scene information from the Bayer thumbnail of the input image, simulating white balance and pre-gamma blocks, generating standard red-green-blue thumbnails, determining representative color channel ratios, merging ideal grid gain and the grid gain of the input image, and applying it to the input image to correct color artifacts.

Benefits of technology

It achieves consistent color rendering of images under different lighting conditions, improves image quality and user satisfaction, and reduces color shadow artifacts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115334261B_ABST
    Figure CN115334261B_ABST
Patent Text Reader

Abstract

A method for performing scene-dependent lens shading correction (SD-LSC) is provided. The method includes: collecting scene information from a Bayer thumbnail of an input image; generating a standard red-green-blue (sRGB) thumbnail by processing the Bayer thumbnail of the input image to simulate white balance (WB) and pre-gamma blocks; determining a representative color channel ratio of the input image based on the scene information and the sRGB thumbnail; determining an ideal grid gain of the input image based on the representative color channel ratio and grid gain; merging the ideal grid gain and the grid gain of the input image to generate a new grid gain; and applying the new grid gain to the input image.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to U.S. Provisional Patent Application Serial No. 63 / 187,013, filed with the U.S. Patent and Trademark Office on May 11, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to reducing image artifacts. Background Technology

[0004] Undesirable lens shading (e.g., vignetting) can occur when the brightness level drops at the periphery of an image compared to the center. Due to insufficient illumination, the corner areas of the lens may appear darker than the center. This unintended darkness is corrected by increasing the brightness level at the corners to compensate for the lack of illumination. Vignetting can be used as an artistic effect, but it is generally considered an artifact that degrades image quality in camera systems.

[0005] Generally, vignetting correction can be divided into two types. The first type is correction based on grid lookup tables (LUTs), and the second type is a functional approximation of the distribution of the correction factor.

[0006] To perform LUT-based correction, a uniform white (or gray) plane is captured in a darkroom to represent the shadow contours of the lens. The shadow contours intersect with a grid section, and a finite number of point values ​​representing the anti-shadow contour gain are stored in the LUT, and for all other points in the frame, the shadow gain is calculated by interpolation with the surrounding grid points.

[0007] Functional approximation correction can be performed in many different ways, such as using polynomials, empirical exponential functions, and / or supercosine functions. Equation (1) below is an example of an approximation function M using a 6th-order polynomial α calculated with a radius r from the center of the image. The type of function depends on the lens design.

[0008] M(r) = 1 + α²r 2 +α4r 4 +α6r 6 ...(1)

[0009] Most Lens Shadow Correction (LSC) solutions utilize factory calibration settings by capturing a uniform gray plane in a darkroom. Factory calibration settings are designed to achieve a static brightness level from the center of the image to every edge or corner. However, when customers shoot real-world scenes under various lighting conditions, factory calibration settings cannot completely eliminate color artifacts. For example, even with factory calibration settings applied, Figure 1The image in the image exhibits visible chromaticity shadow artifacts. In terms of brightness, the image is fine because there is no brightness drop-off throughout the frame. However, in terms of color rendering, the image fails to maintain static red, green, and blue (RGB) proportions, displaying some areas as more green or more red. For example, Figure 1 (a) shows that the image has an unwanted green tint in the central area of ​​the image. Furthermore, Figure 1 (b) shows that the image includes an unwanted red tint in the corners. Additionally, Figure 1 (c) shows that the central area of ​​the image has an unwanted red hue. Summary of the Invention

[0010] This disclosure aims to address the aforementioned problems and shortcomings, and provides at least the following advantages.

[0011] According to one aspect of this disclosure, a method for performing scene-dependent LSC (SD-LSC) includes: collecting scene information from a Bayer thumbnail of an input image; generating a standard red-green-blue (sRGB) thumbnail by processing the Bayer thumbnail of the input image to simulate white balance (WB) and pre-gamma blocks; determining a representative color channel ratio of the input image based on the scene information and the sRGB thumbnail; determining an ideal grid gain of the input image based on the representative color channel ratio and the grid gain of the input image; merging the ideal grid gain and the grid gain of the input image to generate a new grid gain; and applying the new grid gain to the input image.

[0012] According to another aspect of this disclosure, a system for performing SD-LSC includes a memory configured to store instructions and a processor. The processor is configured to execute instructions stored in the memory, which, when executed, cause the processor to collect scene information from a Bayer thumbnail of an input image; generate an sRGB thumbnail by processing the Bayer thumbnail of the input image to simulate WB and pre-gamma blocks; determine a representative color channel ratio of the input image based on the scene information and the sRGB thumbnail; determine an ideal grid gain of the input image based on the representative color channel ratio and the grid gain of the input image; combine the ideal grid gain and the grid gain of the input image to generate a new grid gain; and apply the new grid gain to the input image. Attached Figure Description

[0013] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0014] Figure 1 This shows an image with visible chromaticity shadow artifacts present in the prior art;

[0015] Figure 2 An example of a chromaticity shading artifact and a corresponding color ratio according to an embodiment is shown;

[0016] Figure 3 This is a block diagram illustrating an implementation of SD-LSC according to an embodiment;

[0017] Figure 4 The LSC input is depicted according to an embodiment to obtain a pre-gamma output;

[0018] Figure 5 The image shows an input image with the original gain applied and an input image with the ideal gain applied, according to an embodiment.

[0019] Figure 6 This illustrates applying two different ideal gains to the input image according to an embodiment;

[0020] Figure 7 This is a visualization of the ratio difference based on the embodiment;

[0021] Figure 8 An overall block diagram of the SD-LSC according to an embodiment is shown;

[0022] Figure 9 A flowchart for performing SD-LSC according to an embodiment is shown;

[0023] Figure 10 An example is shown for determining the corresponding Figure 9 The flowchart of the representative color channel ratio in step 903;

[0024] Figure 11 The corresponding embodiment is shown. Figure 9 The flowchart for step 905 in the diagram, which combines the ideal grid gain and the grid gain of the input image; and

[0025] Figure 12 A block diagram of an electronic device in a network environment according to an embodiment is shown. Detailed Implementation

[0026] In the following description, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be noted that although the same elements are shown in different drawings, the same elements will be indicated by the same reference numerals. In the following description, specific details such as detailed configurations and components are provided only to aid in the overall understanding of the embodiments of the present disclosure. Therefore, it will be apparent to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures have been omitted. The terminology described below is defined in consideration of the functions in this disclosure and may vary depending on the user, the user's intent, or habits. Therefore, the definitions of the terms should be determined based on the content throughout this specification.

[0027] This disclosure can have various modifications and embodiments, which are described in detail below with reference to the accompanying drawings. However, it should be understood that this disclosure is not limited to these embodiments, but includes all modifications, equivalents, and alternatives within the scope of this disclosure.

[0028] Although terms including ordinal numbers (such as first, second, etc.) can be used to describe various elements, structural elements are not limited by such terms. The term is used only to distinguish one element from another. For example, without departing from the scope of this disclosure, a first structural element may be referred to as a second structural element. Similarly, a second structural element may also be referred to as a first structural element. As used herein, the term "and / or" includes any and all combinations of one or more associated items.

[0029] The terminology used herein is for describing various embodiments of this disclosure only and is not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In this disclosure, it should be understood that the terms “comprising” or “having” indicate the presence of features, numbers, steps, operations, structural elements, components, or combinations thereof, and do not preclude the presence or possibility of adding one or more other features, numbers, steps, operations, structural elements, components, or combinations thereof.

[0030] Unless otherwise defined, all terms used herein have the same meaning as understood by one of ordinary skill in the art to which this disclosure pertains. Terms such as those defined in general dictionaries shall be interpreted as having the same meaning as in the context of the relevant technical field, and shall not be construed as having an ideal or overly formal meaning unless expressly defined in this disclosure.

[0031] An electronic device according to one embodiment can be one of a variety of types of electronic devices. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computers, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. According to one embodiment of this disclosure, the electronic device is not limited to those described above.

[0032] The terminology used in this disclosure is not intended to limit the disclosure, but rather to include various variations, equivalents, or substitutions of corresponding embodiments. Regarding the description of the drawings, similar reference numerals may be used to refer to similar or related elements. Unless the relevant context clearly indicates otherwise, the singular form of a noun corresponding to an item may include one or more things. As used herein, each of phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include all possible combinations of the items listed together in the corresponding one of the phrases. As used herein, terms such as “first,” “second,” “first,” and “second” may be used to distinguish a corresponding component from another component, but are not intended to limit the components in other respects (e.g., importance or order). It is intended that if an element (e.g., a first element) is referred to (with or without the terms "operably" or "communically") as "coupled to another element (e.g., a second element)," "coupled to another element (e.g., a second element)," "connected to another element (e.g., a second element)," or "connected to another element (e.g., a second element)," it indicates that the element can be coupled to another element directly (e.g., wired), wirelessly, or via a third element.

[0033] As used herein, the term "module" can include units implemented in hardware, software, or firmware, and is used interchangeably with other terms such as "logic," "logic block," "component," and "circuit." A module can be a single integrated component adapted to perform one or more functions, or its smallest unit or component. For example, according to one embodiment, a module can be implemented as an application-specific integrated circuit (ASIC).

[0034] Color rendering is one of the most important factors in the aesthetics of a photograph, but due to the physical limitations of the optical lenses in a camera, some images exhibit inconsistent color rendering. While brightness reduction can be improved through offline factory calibration, controlling chromatic aberration is difficult without real-time adjustments. This disclosure provides embodiments for improving lens shading performance.

[0035] For example, this disclosure utilizes thumbnail statistics generated for each frame in the camera preview. These thumbnail statistics can include all color information of the scene. Based on the scene, an ideal LSC gain can be found and then combined with the original gain to improve image quality.

[0036] Figure 2 An example of chromatic shading artifacts and corresponding color ratios according to an embodiment is shown.

[0037] refer to Figure 2 The R / G ratio is obtained by dividing the average pixel intensity of R by the average pixel intensity of G along the vertical direction of the image. Because... Figure 2 Photos (a) and (b) have more red near the periphery, so the center pixels have a small R / G ratio. Figure 2 The curve (c) is Figure 2 The graph of the R / G ratio in photograph (a) is equivalent to a regular grayscale chart in a darkroom. Figure (c) has a U-shaped pattern; however, the R / G ratio should ideally be represented as a flat line (e.g., a constant R / G ratio). Figure 2 The curve (d) is Figure 2 The R / G ratio of photo (b). Although photo (b) includes many irregularities (e.g., unlike photo (a), photo (b) includes dynamic variations in pixel values), the R / G ratio in graph (d) also resembles a U-shaped pattern, similar to graph (c) (e.g., because it is red along the periphery of photo (b)).

[0038] When adjusting pixel intensity to make curves (c) and (d) flat, chromatic artifacts are reduced or completely eliminated. The algorithm introduced in this disclosure achieves the best possible balance between the R / G ratio and B / G ratio by applying the concept of SD-LSC mesh gain, without degrading the quality of the original color rendering. Conceptually, in Figure 3 The SD-LSC is summarized in the text.

[0039] Figure 3 This is a block diagram illustrating an implementation of SD-LSC according to an embodiment.

[0040] Grid gain (e.g., the gain value assigned to an image region segmented by a grid) can be dynamically updated at each frame based on the captured scene. It can be used to partially control the gain of the region(s) with artifacts, rather than controlling the entire image.

[0041] refer to Figure 3To acquire scene information, SD-LSC can utilize a small thumbnail 301 generated from a previous frame in the preview mode of the camera's image signal processor (ISP). Furthermore, the input grid gain 302 of the current frame can be input to the SD-LSC. Applying SD-LSC, an updated grid gain 303 can be obtained based on the thumbnail 301 of the previous frame and the input grid gain 302 of the current frame. The updated grid gain 303 can then be applied to the image.

[0042] Most LSC techniques are static methods. The gain is allocated according to a predetermined value calibrated in the factory. Although they can reflect a variety of capture conditions (including brightness level, color temperature, or lens position), they have limitations in overcoming color shadow artifacts, which occur unexpectedly more frequently in certain scene modes.

[0043] Because SD-LSC calculates the new gain based on the preview image (e.g., thumbnail 301), it allows for real-time reflection of dynamically changing capture conditions, regardless of the type of scene being presented. The latest updated LSC gain guarantees better color rendering and higher real-world fidelity, thus improving user satisfaction.

[0044] Many different types of LSC models can be used, but these models can often be simplified, as shown in equation (2).

[0045]

[0046] In equation (2), (x, y) is the pixel position, and c is the color channel. The gain G consists of factory-calibrated values ​​to compensate for static brightness levels and is independent of pixel position. Because this LSC model operates independently without the interaction between color channels, it is impossible to detect color deviations that produce color shading.

[0047] Most camera ISPs include the ability to apply statistical algorithms for autofocus, auto exposure, and auto white balance to estimate ideal color rendering conditions based on statistical thumbnails of preview frames. This is known as the 3A operation. The result of the 3A operation can be referred to as "scene information." The 3A operation and thumbnail creation are performed in the Bayer domain for real-time processing. For example, applying autofocus in the Bayer domain can be referred to as applying a "3A statistical algorithm." Similarly, applying auto exposure and / or auto white balance in the Bayer domain can each be referred to as applying a "3A statistical algorithm."

[0048] When the ISP generates the final sRGB output (e.g., a Joint Group of Image Experts (JPEG) file), the user will observe color shading. One aspect of SD-LSC is to use thumbnails generated from previous frames to predict chromaticity shading. However, some artifacts that are visible in the Bayer domain and when visible in the sRGB domain may not be observable in the thumbnail. To determine how RGB colors are distributed, SD-LSC performs a simple sRGB process by simulating WB and pregamma blocks. The combination of LSC, WB, and pregamma can be represented as shown in equation (3).

[0049] I o (x,y,c)=(W(c)G(x,y,c)I i (x,y,c)) 1 / 2.2 ...(3)

[0050] In equation (3), I i It is an LSC input, I o is the pre-gamma output, W is the WB gain, and the pre-gamma is assumed to be the commonly used power of 1 / 2.2.

[0051] Figure 4 The process according to an embodiment is described to obtain a pre-gamma output I. o LSC input I i .

[0052] refer to Figure 4 Bayer patterns can be applied to input I i After performing LSC and WB, a color-balanced image can be obtained. Furthermore, after performing pre-gamma, a brightness-balanced image can be obtained as the pre-gamma output. o .

[0053] Another aspect of this disclosure is the implementation of HW ISP, in which fixed-point (FXP) operations and displacement scaling are applied. FXP operation processing can be applied to modify equation (3), as shown in equation (4) below.

[0054]

[0055] In equation (4), d0 is the pre-gamma displacement, d1 is the WB displacement, and d2 is the LSC displacement. Along with displacement scaling, each processing function I... i The elements of G(x,y,c), W(c) and G(x,y,c) are integers.

[0056] Obtaining pre-gamma output I o Then, the grid gain value G(x,y,c) can be derived in reverse using equation (5).

[0057]

[0058] Now suppose we have a pre-gamma output value that is completely free of chroma shading artifacts; this can be called "ideal gain" or "ideal mesh gain." Figure 2 In the images (a) and (b), there are chromaticity shading artifacts, and therefore uneven R / G curves (c) and B / G curves (d), respectively. An image with ideal gain will make the R / G ratio or B / G ratio always constant, regardless of the pixel position across the entire region of the image. This represents the ideal pre-gamma output. The ideal red and blue channel outputs relative to the green channel can be considered as the same ratio as shown in equation (6), where K represents a constant.

[0059]

[0060] In an image with ideal gain, the pixel values ​​are as shown in equation (7).

[0061]

[0062] Therefore, by substituting equation (7) into equation (4), the ideal grid gain value can be estimated as shown in equation (8).

[0063]

[0064] Once the constant K is determined, the ideal mesh gain value that makes the image completely free of chromaticity artifacts can be calculated.

[0065] An image with ideal gain (e.g., an ideal image) is not the final goal. More specifically, if ideal gain is applied as is, it will only represent a color ratio applied to the entire area of ​​the image, losing color information.

[0066] Figure 5 The diagram shows an input image with the original gain applied and an input image with the ideal gain applied, according to an embodiment.

[0067] To preserve color information, the final output gain G new It is determined by combining (or merging) the input gain and the ideal gain, as shown in Equation (9).

[0068]

[0069] Although the method used to determine β will be described in more detail below, the overall idea is that more input gain G is needed in richly colored regions, and a more desirable gain is needed in regions where chromatic shading is visible (e.g., less colorful regions). In other words, the input gain G is applied to some regions of the image (e.g., regions rich in color), and the ideal gain... It is applied to other areas of the image (e.g., areas with less color).

[0070] It is important to choose constants K(R) and K(B) that represent the R / G ratio and B / G ratio of the entire image.

[0071] Figure 6 The illustration shows, according to an embodiment, the application of two different ideal gains to the input image.

[0072] refer to Figure 6 The input image is shown on the left side of the diagram. If a random ratio is chosen, such as... Figure 6 In the case of #1 (corresponding to (R,G,B)=(177,122,141)), the result will have a large color shift, such as Figure 6 Image (a) (which appears to have a heavy red hue) shows no chromaticity shadow artifacts even under a constant color ratio. However, the choice Figure 6 #2 (corresponding to (R,G,B)=(113,118,117)) is a better representative ratio because it has no significant impact on the overall offset in the color rendering of the original image, such as Figure 6 As demonstrated by image (b) in the image.

[0073] For this reason, it is preferable to select the R / G and B / G ratios that are present on most pixels in the image. In statistics, the mode of a set of data values ​​is the most frequently occurring value. Therefore, the mode of the R / G and B / G ratios should be selected as the representative of the pre-gamma output in equation (4) derived from the thumbnail. Equation (10) represents the ratio relative to the green channel.

[0074]

[0075] and It can be any decimal number, but the application is designed for FXP operations, which means... and It is a discrete integer. If the thumbnail width is P and the height is Q, then by counting the pregamma output with a total number of samples of P×Q... The quantity can generate information about and A two-dimensional (2D) histogram. This can be represented as a 2D histogram of the R / G ratio and the B / G ratio. Then, as shown in equation (11), the mode values ​​of the R and B channels become The index of the maximum value.

[0076]

[0077] Because the number of samples P×Q may be too sparse to generate a 2D histogram, and because the thumbnail is a miniature version of the full frame, the maximum value may not be sufficient to reliably represent the entire image. Therefore, as shown in equation (12), a local average can be used instead. To obtain the modal values ​​of the R and B channels.

[0078]

[0079] Therefore, the representativeness ratio (K(R),K(B)) can be obtained through equation (13).

[0080]

[0081] As described above, the ideal gain is applied to certain parts of the image, and the input gain is applied to other parts. The final gain value applied to the image is determined by combining the ideal gain and the input gain for the individual regions of the image. Color-rich regions of the image (e.g., regions of the image with a relatively high difference in chromaticity values ​​relative to the average chromaticity values ​​of the image), where there are no lens shading artifacts, do not need to use the ideal image gain and can retain the input gain based on the original pixel values. Chromaticity shading artifacts are more likely to be visible in regions where the color ratio of the input image differs slightly from the ideal ratio. Therefore, applying the ideal image gain is more likely to be used for less color-rich regions of the image (e.g., regions of the image with a relatively low difference in chromaticity values ​​relative to the average chromaticity values ​​of the image).

[0082] The "merging" of ideal and input gains (e.g., applying the ideal gain to some regions of an image and the input gain to other regions of the image) can be performed in two steps: the difference step and the gain difference step.

[0083] As shown in equation (14), the ratio difference d r (x,y) can be used as weights to combine the ideal and the input gain.

[0084]

[0085] Figure 7 The difference d is based on the embodiment. r Visualization of (x,y). Comparison of differences d r (x,y) can be the ratio difference in the color channels.

[0086] refer to Figure 7 Example image (a) shows the Macbeth color test area and green plants. The corresponding difference d r(x, y) (a linear combination of input gain and ideal gain) in Figure 7 As shown in (b). Based on equation (14), the ideal and original gains are combined, making it more likely that the ideal mesh gain will be used in dark / black regions and more likely that the original gain will be used in white / bright regions. The merging process introduced by equation (9) becomes the following equations (15) and (16):

[0087] d r (x, y) := max(min(d r (x, y), d r_max ), d r_min )-d r_min ...(15)

[0088]

[0089] In equations (15) and (16), d r_max and d r_min These are upper and lower limits, meaning that if the ratio exceeds the corresponding upper or lower limit, only the input gain or only the ideal gain is used.

[0090] In addition to the comparison step, a gain difference step can also be included to combine the ideal gain with the input gain. If G ratio Values ​​that differ too much from the original input gain, even with small ratio differences, can be the root cause of unwanted color biases (e.g., color shift artifacts) in the final sRGB output image. Many images can be merged using ratio differences, but some may require additional merging techniques to eliminate other side effects. For this purpose, when the new gain G... ratio When the value differs too much from the input gain, the process of continuing to apply the input gain can be used according to equations (17)-(19).

[0091] d g (x, y, c) = |G ratio (x,y,c)-G(x,y,c)| ...(17)

[0092] d g (x, y, c) := max(min(d g (x, y, c), d g_max ), d g_min )-d g_min ...(18)

[0093] G out (x, y, c) = (G(x, y, c)d g (x, y, c) + G ratio (x, y, c)(d) g_max -dg_min -d g (x, y, c))) / (d g_max -d g_min ...(19)

[0094] Therefore, G out (x,y,c) can be the final gain output in the above two-step merging process.

[0095] Figure 8 An overall block diagram of the SD-LSC according to an embodiment is shown.

[0096] refer to Figure 8 In (a), a Bayer thumbnail 801 and an input grid gain 802 are provided as input. The Bayer thumbnail 801 is processed by applying WB and pre-gamma, and representativeness ratios K(R) and K(B) are calculated for the processed thumbnail. Furthermore, the size of the Bayer thumbnail 801 is adjusted (e.g., downsampled) before applying WB and pre-gamma to determine the ideal grid gain, because generally, the grid gain size (W×H) is smaller than the thumbnail size (P×Q). Alternatively, the grid gain size can be used instead of the size of the upsampled thumbnail 801 (e.g., P×Q), which can be used to generate better quality images. However, when low computational complexity is preferred, such as when using smartphone preview mode or video recording requiring at least 30 frames per second (fps) of real-time stable processing, thumbnail downsampling is a better approach than upsampling. Once the ideal grid gain is obtained, merging the ideal grid gain with the input grid gain is performed to obtain a new (final) grid gain.

[0097] refer to Figure 8 Section (b) will describe in detail the steps for calculating the representativeness ratios K(R) and K(B). That is, in order to calculate the representativeness ratios K(R) and K(B), the processed thumbnails are used to calculate the R, G, and B ratios. For example, as explained in the above reference equation (10), the R / G and B / G ratios (e.g., the mode) most likely to appear on the majority of pixels are selected for calculation. and Then, as explained in the above reference equation (11), using and To generate 2D histograms Furthermore, as explained in the above reference equation (12), the local average value of the 2D histogram is determined. And as explained in the above reference equation (13), the local average value The maximum ratio is used to calculate the representativeness ratios K(R), K(B)).

[0098] refer to Figure 8Section (c) will describe in detail the steps for combining the ideal gain with the input gain. As mentioned above, one or both of the ratio difference and / or gain difference are used to combine the ideal gain with the input gain.

[0099] As shown in equation (14), the ratio difference d is used. r (x, y) are used as weights for combining the ideal gain and the input gain. If the difference d r (x, y) exceeds the corresponding upper limit d r_max Or lower limit d r_min If so, then only the input gain or only the ideal gain is used.

[0100] Besides the comparison difference step, a gain difference step can also be used to combine the ideal gain with the input gain. If based on the comparison difference d... r The adjusted gain value G of (x,y) ratio If the difference from the original input gain G(x,y,c) is too large, then as explained in the above reference equations (17)-(19), the original input gain value can be used instead of the adjusted gain value.

[0101] Figure 9 A flowchart for performing SD-LSC according to an embodiment is shown.

[0102] Figure 9 The steps shown can be performed in an alternating order and / or some steps can be omitted, and additional steps can be added. Furthermore, Figure 9 The steps shown can be stored as instructions and executed by the processor.

[0103] refer to Figure 9 In step 901, scene information is collected from the Bayer thumbnail of the input image. For example, as described above, scene information can be collected by applying at least one 3A statistical algorithm to the Bayer thumbnail.

[0104] In step 902, an sRGB thumbnail is generated by processing the Bayer thumbnail to simulate WB and / or pre-gamma blocks. For example, after performing WB, a color-balanced image can be obtained. Furthermore, after performing pre-gamma, a brightness-balanced image can be obtained as the pre-gamma output.

[0105] In step 903, a representative color channel ratio of the input image is determined (e.g., calculated or computed) based on scene information and the sRGB thumbnail. The following will refer to... Figure 10 Further discussion on step 903.

[0106] In step 904, the ideal grid gain of the input image is determined based on the representative color channel ratio and the grid gain of the input image. For example, the ideal grid gain can be calculated using equation (8) above, and the grid gain of the input image can be provided as input.

[0107] In step 905, the ideal grid gain and the grid gain of the input image are combined to generate a new grid gain. The following will refer to... Figure 11 Further discussion on step 905.

[0108] In step 905, the new grid gain is applied to the image (e.g., the input image).

[0109] Figure 10 An example is shown for determining the corresponding Figure 9 The flowchart of the representative color channel ratio in step 903.

[0110] Figure 10 The steps shown can be performed in an alternating order and / or some steps can be omitted, and additional steps can be added. Furthermore, Figure 10 The steps shown can be stored as instructions and executed by the processor.

[0111] refer to Figure 10 Provide the thumbnail to be processed as input. For example... Figure 4 As shown, it can be assumed that the processed thumbnail has undergone WB and / or pre-gamma processing, and can correspond to the pre-gamma output I. o .

[0112] In step 1001, the R, G, B ratio is calculated. For example, as explained above with reference to equation (10), the color ratio most likely to appear on most pixels (e.g., the mode) can be calculated.

[0113] In step 1002, a 2D histogram is calculated. For example, a 2D histogram of color ratios (e.g., R / G ratio, B / G ratio, B / R ratio, G / R ratio, R / B ratio, or G / B ratio) can be relative to the colors across the entire image (e.g., and It is calculated using the average value of ).

[0114] In step 1003, local averages are calculated using a 2D histogram. Furthermore, as explained above with reference to equation (13), in step 1004, representativeness ratios K(R) and K(B) (e.g., representative color ratios) are calculated using the maximum ratio of local averages.

[0115] In addition, the ideal gain can be calculated using the representative ratios K(R) and K(B).

[0116] Figure 11 The corresponding embodiment is shown. Figure 9 The flowchart for step 905 in the diagram, which is used to combine the ideal grid gain and the grid gain of the input image.

[0117] Figure 11 The steps shown can be performed in an alternating order and / or some steps can be omitted, and additional steps can be added. Furthermore, Figure 11 The steps shown can be stored as instructions and executed by the processor.

[0118] Two steps, namely the comparison-difference step and the gain-difference step, can be used to combine the ideal and input gains (e.g., applying the ideal gain to some regions of the image and applying the input gain to other regions of the image). Steps 1101-1102 correspond to the comparison-difference step, and steps 1103-1104 correspond to the gain-difference step.

[0119] refer to Figure 11 In step 1101, the ratio difference is calculated. The ratio difference can be a linear combination of the input gain and the ideal gain. In step 1102, the ratio difference is combined. For example, the ideal gain and the input gain can be combined based on the above equation (14) so ​​that the adjusted gain value can be used based on the ratio difference.

[0120] In step 1103, the gain difference is calculated. If the combined gain value (calculated in step 1102) differs too much from the original input gain, then in step 1104, the gain difference is combined so that the original input gain value can be used instead of the adjusted gain value (as explained above in reference equations (17)-(19)).

[0121] Figure 12 An electronic device in a network environment according to an embodiment is shown.

[0122] refer to Figure 12In network environment 1200, electronic device 1201 (e.g., a mobile terminal including GPS functionality) can communicate with electronic device 1202 via a first network 1298 (e.g., a short-range wireless communication network), or with electronic device 1204 or server 1208 via a second network 1299 (e.g., a long-range wireless communication network). Electronic device 1201 can communicate with electronic device 1204 via server 1208. Electronic device 1201 may include processor 1220, memory 1230, input device 1250, sound output device 1255, display device 1260, audio module 1270, sensor module 1276, interface 1277, haptic module 1279, camera module 1280, power management module 1288, battery 1289, communication module 1290, subscriber identification module (SIM) 1296, or antenna module 1297 including a GNSS antenna. In one embodiment, at least one component (e.g., display device 1260 or camera module 1280) may be omitted from electronic device 1201, or one or more other components may be added to electronic device 1201. In one embodiment, some components may be implemented as a single integrated circuit (IC). For example, sensor module 1276 (e.g., fingerprint sensor, iris sensor, or illuminance sensor) may be embedded in display device 1260 (e.g., display).

[0123] Processor 1220 can run, for example, software (e.g., program 1240) to control at least one other component (e.g., hardware or software component) of electronic device 1201 coupled to processor 1220, and can perform various data processing or calculations. As at least part of the data processing or calculations, processor 1220 can load commands or data received from another component (e.g., sensor module 1276 or communication module 1290) into volatile memory 1232, process the commands or data stored in volatile memory 1232, and store the resulting data in non-volatile memory 1234. Processor 1220 may include a main processor 1221 (e.g., a central processing unit (CPU) or application processor) and an auxiliary processor 1223 (e.g., a graphics processing unit (GPU), image signal processor (ISP), sensor hub processor, or communication processor (CP)), which may operate independently of or in conjunction with the main processor 1221. Additionally or optionally, the auxiliary processor 1223 may be adapted to consume less power than the main processor 1221 or to perform specific functions. The auxiliary processor 1223 may be implemented separately from or as part of the main processor 1221.

[0124] When the main processor 1221 is inactive (e.g., in sleep mode), the auxiliary processor 1223 may take over from the main processor 1221. Alternatively, when the main processor 1221 is active (e.g., running an application), the auxiliary processor 1223 may work with the main processor 1221 to control at least some functions or states associated with at least one component of the electronic device 1201 (e.g., display device 1260, sensor module 1276, or communication module 1290). According to one embodiment, the auxiliary processor 1223 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 1280 or communication module 1290) functionally associated with the auxiliary processor 1223.

[0125] Memory 1230 may store various types of data used by at least one component of electronic device 1201 (e.g., processor 1220 or sensor module 1276). The various types of data may include, for example, input or output data of software (e.g., program 1240) and associated commands. Memory 1230 may include volatile memory 1232 or non-volatile memory 1234.

[0126] Program 1240 may be stored as software in memory 1230 and may include, for example, an operating system (OS) 1242, middleware 1244, or application 1246.

[0127] Input device 1250 can receive commands or data from outside electronic device 1201 (e.g., a user) that will be used by another component of electronic device 1201 (e.g., processor 1220). Input device 1250 may include, for example, a microphone, mouse, or keyboard.

[0128] The sound output device 1255 can output sound signals to the outside of the electronic device 1201. The sound output device 1255 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or recording, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0129] Display device 1260 can visually provide information to the outside of electronic device 1201 (e.g., to a user). Display device 1260 may include, for example, a display, a holographic device, or a projector, and control circuitry controlling a corresponding one of the display, holographic device, and projector. According to one embodiment, display device 1260 may include touch circuitry adapted to detect touch, or sensor circuitry adapted to measure the intensity of the force caused by touch (e.g., a pressure sensor).

[0130] The audio module 1270 can convert sound into electrical signals and vice versa. According to one embodiment, the audio module 1270 can obtain sound via input device 1250, or output sound via sound output device 1255 or headphones of external electronic device 1202 directly (e.g., wired) or wirelessly coupled to electronic device 1201.

[0131] Sensor module 1276 can detect the operating state of electronic device 1201 (e.g., power or temperature) or the environmental state outside electronic device 1201 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. Sensor module 1276 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biosensor, temperature sensor, humidity sensor, or illuminance sensor.

[0132] Interface 1277 may support one or more specified protocols for direct (e.g., wired) or wireless coupling of electronic device 1201 to external electronic device 1202. According to one embodiment, interface 1277 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0133] The connection terminal 1278 may include a connector that can be physically connected to an external electronic device 1202 via its electronic device 1201. According to one embodiment, the connection terminal 1278 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0134] The haptic module 1279 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) and electrical stimuli that can be recognized by a user via touch or kinesthesia. According to one embodiment, the haptic module 1279 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0135] Camera module 1280 can capture still images or moving images. According to one embodiment, camera module 1280 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0136] The power management module 1288 can manage the power supplied to the electronic device 1201. The power management module 1288 can be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0137] Battery 1289 can supply power to at least one component of electronic device 1201. According to one embodiment, battery 1289 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0138] Communication module 1290 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 1201 and external electronic devices (e.g., electronic device 1202, electronic device 1204, or server 1208), and perform communication via the established communication channel. Communication module 1290 may include one or more communication processors that can operate independently of processor 1220 (e.g., application processor) and support direct (e.g., wired) or wireless communication. According to one embodiment, communication module 1290 may include wireless communication module 1292 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 1294 (e.g., local area network (LAN) communication module or power line communication (PLC) module). One of these communication modules can communicate with an external electronic device via a first network 1298 (e.g., a short-range communication network, such as Bluetooth™, Wi-Fi Direct, or Infrared Data Association (IrDA) standards) or a second network 1299 (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN)). These different types of communication modules can be implemented as a single component (e.g., a single IC) or as multiple components that are separate from each other (e.g., multiple ICs). The wireless communication module 1292 can use subscriber information (e.g., International Mobile Subscriber Identity (IMSI)) stored in the subscriber identification module 1296 to identify and authenticate the electronic device 1201 in the communication network such as the first network 1298 or the second network 1299.

[0139] Antenna module 1297 can transmit signals or power to or from the outside of electronic device 1201 (e.g., external electronic device). According to one embodiment, antenna module 1297 may include one or more antennas, and thus, for example, communication module 1290 (e.g., wireless communication module 1292) can select at least one antenna suitable for a communication scheme used in a communication network such as a first network 1298 or a second network 1299. Signals or power can then be transmitted or received between communication module 1290 and external electronic device via the selected at least one antenna.

[0140] At least some of the aforementioned components may be coupled to each other and communicate signals (e.g., commands or data) between them via a peripheral communication scheme (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).

[0141] According to one embodiment, commands or data can be sent or received between electronic device 1201 and external electronic device 1204 via server 1208 coupled to a second network 1299. Each of electronic devices 1202 and 1204 can be a device of the same or different type as electronic device 1201. All or some operations to be performed on electronic device 1201 can be performed on one or more external electronic devices 1202, 1204, or 1208. For example, if electronic device 1201 is required to automatically perform a function or service, or in response to a request from a user or another device, instead of performing the function or service, electronic device 1201 can request one or more external electronic devices to perform at least a portion of the function or service. The one or more external electronic devices receiving the request can perform at least a portion of the requested function or service, or additional functions or services related to the request, and transmit the result of the performance to electronic device 1201. Electronic device 1201 can provide the result, with or without further processing, as at least part of a response to the request. For this purpose, technologies such as cloud computing, distributed computing, or client-server computing can be used.

[0142] One embodiment can be implemented as software (e.g., program 1240) including one or more instructions stored in a machine-readable storage medium (e.g., internal memory 1236 or external memory 1238). For example, a processor of electronic device 1201 can invoke at least one of the one or more instructions stored in the storage medium and execute the instructions with or without one or more other components under the control of the processor. Thus, the machine can be operated to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between a location where data is semi-permanently stored in the storage medium and a location where data is temporarily stored in the storage medium.

[0143] According to one embodiment, the methods disclosed herein can be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM The computer program product may be distributed online (e.g., downloaded or uploaded) or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as the memory of a manufacturer's server, an app store's server, or a relay server.

[0144] According to one embodiment, each of the above-described components (e.g., a module or program) may include a single entity or multiple entities. One or more of the above-described components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component can still perform one or more functions of each of the multiple components in the same or similar manner as it was performed by a corresponding component among the multiple components before integration. Operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be run in a different order or omitted, or one or more other operations may be added.

[0145] Although certain embodiments of this disclosure have been described in the detailed description thereof, modifications may be made in various forms without departing from the scope of this disclosure. Therefore, the scope of this disclosure should not be determined solely based on the described embodiments, but rather on the appended claims and their equivalents.

Claims

1. A method for performing scene-dependent lens shading correction (SD-LSC), the method comprising: Collect scene information from Bayer thumbnails of the input image; Standard red-green-blue (sRGB) thumbnails are generated by processing the Bayer thumbnails of the input image to simulate white balance (WB) and pre-gamma blocks; The representative color channel ratio of the input image is determined based on scene information and sRGB thumbnails; The ideal grid gain of the input image is determined based on the representative color channel ratio and grid gain of the input image. The ideal grid gain and grid gain of the input image are merged so that the grid gain of the input image is applied to the color-rich regions of the input image and the ideal grid gain is applied to the color-inferior regions of the input image to generate a new grid gain; as well as Apply the new grid gain to the input image.

2. The method according to claim 1, wherein, Generate Bayer thumbnails from previous frames of the input image.

3. The method according to claim 2, wherein, The grid gain of the input image is based on the current frame of the input image.

4. The method of claim 1, further comprising adjusting the size of the sRGB thumbnail before determining the ideal grid gain.

5. The method according to claim 1, wherein, Use at least one two-dimensional (2D) histogram of R / G ratio, B / G ratio, B / R ratio, G / R ratio, R / B ratio, or G / B ratio to determine the representative color channel ratio.

6. The method according to claim 5, wherein, The mode value of at least one of the R channel, B channel, or G channel is used to determine the maximum value of the at least one 2D histogram.

7. The method according to claim 6, wherein, The mode value of at least one of the R, B, or G channels is determined using the local mean of the 2D histogram.

8. The method according to claim 1, wherein, The ideal grid gain and grid gain of the merged input image include applying upper and lower bounds to the ratio of the ideal grid gain and grid gain of the input image to the linear combination of the color channels.

9. The method according to claim 8, wherein, The merging of the ideal grid gain and grid gain of the input image also includes applying an upper and lower bound to the gain difference between the ideal grid gain and the grid gain of the input image.

10. A system for performing scene-dependent lens shading correction (SD-LSC), the system comprising: The memory is configured to store instructions; as well as A processor is configured to execute instructions stored in memory, which, when executed, cause the processor to: Collect scene information from the Bayer thumbnail of the input image; Standard red-green-blue (sRGB) thumbnails are generated by processing the Bayer thumbnails of the input image to simulate white balance (WB) and pre-gamma blocks; The representative color channel ratio of the input image is determined based on scene information and sRGB thumbnails; The ideal grid gain of the input image is determined based on the representative color channel ratio and the grid gain of the input image; The ideal grid gain and grid gain of the input image are combined so that the input grid gain of the input image is applied to the color-rich regions of the input image and the ideal grid gain is applied to the color-inferior regions of the input image to generate a new grid gain; as well as Apply the new grid gain to the input image.

11. The system according to claim 10, wherein, Generate Bayer thumbnails from previous frames of the input image.

12. The system according to claim 11, wherein, The grid gain of the input image is based on the current frame of the input image.

13. The system according to claim 10, wherein, Adjust the size of the sRGB thumbnail before calculating the ideal grid gain.

14. The system according to claim 10, wherein, The representative color channel ratios are calculated using at least one two-dimensional (2D) histogram of R / G ratio, B / G ratio, B / R ratio, G / R ratio, R / B ratio, or G / B ratio.

15. The system according to claim 14, wherein, The mode value of at least one of the R channel, B channel, or G channel is used to calculate the maximum value of the at least one 2D histogram.

16. The system according to claim 15, wherein, The mode value of at least one of the R, B, or G channels is calculated using the local mean of the 2D histogram.

17. The system according to claim 10, wherein, The ideal grid gain and grid gain of the merged input image include applying upper and lower bounds to the ratio of the ideal grid gain and grid gain of the input image to the linear combination of the color channels.

18. The system according to claim 17, wherein, The merging of the ideal grid gain and grid gain of the input image also includes applying an upper and lower bound to the gain difference between the ideal grid gain and the grid gain of the input image.