Content-Adaptive Lens Shadow Correction Method and Apparatus

Abstract

Methods and apparatus for providing adaptive lens shading correction in at least a portion of a shadowed image frame: The method performs luminance lens shading correction on a portion of the shadowed image frame and detects color-flat regions in the shadowed image. The method and apparatus generate a frequency-based color shading distribution map including color shading parameters, which include amplitude and phase parameters corresponding to the hue distribution from experimental shadow data, and generate a shadow-free image by performing color shading correction on the detected color-flat regions of the shadowed image frame using the color shading parameters.

Description

Background Technology

[0001] Lens shading correction (LSC) is used to correct image shading caused by various factors related to the photodetector during the production of digital images or other image capture devices. Shading, also known as vignetting, refers to a decrease in image brightness and saturation towards the periphery of the image compared to the center. Lens shading is caused by many factors, including but not limited to lens aperture, lens barrel, sensor principal ray angle, and others. Additionally, light transmittance varies with incident wavelength, caused by refraction, sensor sensitivity, pixel crosstalk, and other factors. Thus, both brightness shading and color shading can alter image quality. Typically, photodetectors include different color component sensors, such as separate red, green, and blue pixel sensors. In some implementations, the lens shading correction circuit employs a lookup table with a stored correction value for each pixel of the photodetector. In some implementations, an RGB (R / G / G / B) photodetector is used in the image capture device.

[0002] Lens shading can be a difficult problem to correct because shadows can vary from image sensor unit to image sensor unit due to sensor variations, lens variations, component variations, aging, and other factors. Additionally, shadows vary depending on the scene, such as the wavelength of light sources and objects within the scene. Point-to-point real-time video systems using mobile phones, laptops, and video teleconferencing systems can experience significant quality degradation when a user views a display showing live video being captured by one or more image capture devices at the other end of a video conference.

[0003] Some solutions employ dynamic lens shading correction, which relies on an automatic white balance mechanism for color temperature prediction. However, this can be insufficiently precise and may fail to address issues where colors differ only in wavelength. Other solutions, sometimes referred to as adaptive lens shading correction, attempt to predict color temperature based on the image content of the image frame. Such techniques offer high performance in eliminating lens shading, but their implementation can be complex. Attached Figure Description

[0004] The implementation will be more readily understood when viewed in conjunction with the following figures, as described below, wherein the same reference numerals denote the same elements, and wherein:

[0005] Figure 1 This is a block diagram illustrating an example of an apparatus for providing adaptive lens shading correction according to the present disclosure;

[0006] Figure 2 It is a graphical representation of the tonal distribution of test images captured by one or more image capture devices from experimental shadow data;

[0007] Figure 3 This is a flowchart illustrating a method for providing adaptive lens shading correction according to an example presented in this disclosure;

[0008] Figure 4 This is a block diagram illustrating an adaptive lens shading correction circuit with frequency-based color correction, according to an example presented in this disclosure;

[0009] Figure 5 This is a flowchart illustrating a method for providing adaptive lens shading correction according to an example presented in this disclosure;

[0010] Figure 6 This is a flowchart illustrating a method for providing adaptive lens shading correction according to an example presented in this disclosure;

[0011] Figure 7 This is a flowchart illustrating a method for providing adaptive lens shading correction according to an example proposed in this disclosure; and

[0012] Figure 8 This is a flowchart illustrating a method for providing adaptive lens shading correction according to an example presented in this disclosure.

[0013] In the following description, the same reference numerals are used in different figures to denote similar or identical items. Unless otherwise stated, the word “coupled” and its associated verb form include both direct connection and indirect electrical connection in a manner known in the art, and unless otherwise stated, any description of a direct connection implies an alternative implementation using a suitable form of indirect electrical connection. Detailed Implementation

[0014] The techniques disclosed herein, including methods and apparatus, achieve efficient and cost-effective adaptive lens shading correction by using frequency-based color shading distribution maps. By correcting for luminance and color shading caused by the image capture device, adaptive lens shading correction can be performed on variations in image content within a video. In one example, color shading is corrected using a frequency-based color shading distribution map based on stored hue distribution data for each color component. Extremely low-frequency signals in hue-flat regions of the shading image are analyzed, while high-frequency signals are filtered out.

[0015] In some implementations, the apparatus and methods for providing adaptive lens shading correction employ a frequency-based color shading distribution map for each color component, including color shading parameters such as amplitude and phase parameters corresponding to the hue distribution of experimental shading data based on test images captured by one or more image capture devices. In some implementations, the frequency-based color shading distribution map uses a first-order harmonic frequency distribution as the basis for determining shading correction parameters for content in the shading image to correct color shading in the shading image. In some embodiments, the method and apparatus store statistical data on the hue distribution of different color components used to generate the frequency-based color shading distribution map. In some examples, this includes storing hue distribution data for each of a plurality of color components corresponding to one or more image capture devices, and using, for example, the first harmonic of the distribution to extract the frequency distribution of the hue distribution to determine the shading distribution map for one or more image capture devices.

[0016] In some examples, the method and apparatus remove luminance shading (also known as photoluminescence shading) and perform downsampling on pixel blocks of the shadowed image as part of a color shading correction operation. For color shading correction, in some examples, the method and apparatus detect color-flat regions of the shadowed image pixel by pixel based on a hue flatness threshold to determine color (hue) flat regions of the shadowed image frame. Color-unflattened regions, also known as hue-uniform regions, are extracted. Extremely low-frequency signals in the hue-flat regions are analyzed while high-frequency signals are filtered out. In some examples, the method includes: calculating frequency-based color shading distribution map parameters, in one example including amplitude and phase parameters from experimental shadow data; and using the color shading parameters to generate correction parameters to remove shading from color-flat regions in the shadowed image and produce a shadow-free image. The shadow-free image can be compressed and sent to another process, device, network, or, in some examples, for display. In some implementations, the shading correction operation is performed on a network such as a cloud-based platform.

[0017] According to some embodiments, a method for providing adaptive lens shading correction in at least a portion of a shadowed image frame includes performing luminance lens shading correction on a portion of the shadowed image frame; detecting color-flat regions in the shadowed image; generating a frequency-based color shading distribution map including color shading parameters, the color shading parameters including amplitude and phase parameters corresponding to a hue distribution from experimental shadow data based on test images captured by one or more image capture devices; and generating a shadow-free image by performing color shading correction on the detected color-flat regions of the shadowed image frame using the color shading parameters including the amplitude and phase parameters corresponding to the hue distribution from the experimental shadow data. The generation of the shadow distribution map can be done online or offline.

[0018] In some examples, the tone distribution includes stored tone distribution data for each of a plurality of color components corresponding to one or more image capture devices, and wherein detecting color flat regions in a shadow image includes downsampling a shadow image frame to produce one or more downsampled pixel blocks of the shadow image frame.

[0019] In some examples, the method includes converting the hue in the color-flat region from the spatial domain to the frequency domain, determining color shading correction parameters based on the color shading model parameters of the generated frequency-based color shading distribution map, and minimizing the sum of hue variations based on each pixel block.

[0020] In some examples, the generated frequency-based color shading distribution map f(x,y) is represented as:

[0021] f(x,y)=g(x)·g(y)

[0022]

[0023] Where f(x,y) is the color shading distribution, g(x) and g(y) represent the horizontal and vertical directions, and A is the amplitude. W is the phase, W is the image width, and H is the image height.

[0024] In some examples, converting hues in color-flat regions includes converting hue signals detected in the color-flat regions of pixel blocks in both horizontal and vertical directions into the frequency domain, and extracting the low-frequency components of the conversion in each direction using a low-pass filter. The method includes converting each obtained low-frequency component into a spatial distribution and determining color shading correction parameters by generating a color shading correction table for adjusting gain control of image correction circuitry to produce a shading-free image.

[0025] In some examples, performing luminance lens shading correction involves generating luminance correction table information from shading image frames using a pre-calibrated luminance shading table based on experimental testing of one or more image capture devices, and applying the luminance correction table information to correct luminance shading.

[0026] According to some embodiments, an apparatus for providing adaptive lens shading correction in at least a portion of a shadowed image frame includes a brightness correction circuit that performs brightness lens shading correction on at least a portion of the shadowed image frame. The apparatus includes: a tone unevenness removal detection circuit that detects color-flat regions in the shadowed image; and a frequency-based color correction circuit that generates a frequency-based color shading distribution map including color shading parameters based on a test image captured by one or more image capture devices, the color shading parameters including amplitude and phase parameters corresponding to the tone distribution from experimental shadow data. The frequency-based color correction circuit generates a shadow-free image by performing color shading correction on the detected color-flat regions of the shadowed image frame using the color shading parameters, the color shading parameters including amplitude and phase parameters corresponding to the tone distribution from the experimental shadow data.

[0027] In some examples, the device includes a memory that stores data representing the tonal distribution, which includes tonal distribution data corresponding to each of a plurality of color components of one or more image capture devices.

[0028] In some examples, the color unevenness removal detection circuit detects color flat areas in a shadow image by downsampling the shadow image frame to produce one or more downsampled pixel blocks of the shadow image frame.

[0029] In some examples, the frequency-based color correction circuit converts the hue in the color flat region from the spatial domain to the frequency domain, and determines the color shading correction parameters based on the color shading model parameters of the generated frequency-based color shading distribution map by minimizing the sum of hue variations.

[0030] In some examples, the frequency-based color correction circuit uses the generated frequency-based color shading distribution map f(x,y) to produce shading model parameters:

[0031] f(x,y)=g(x)·g(y)

[0032]

[0033] Where f(x,y) is the color shading distribution, g(x) and g(y) represent the horizontal and vertical directions, and A is the amplitude. W is the phase, W is the image width, and H is the image height.

[0034] In some examples, the frequency-based color correction circuit converts the hue signals detected in the color-flat regions of the pixel blocks in both the horizontal and vertical directions into the frequency domain, extracts the low-frequency components of the conversion in each direction using a low-pass filter, converts each obtained low-frequency component into a spatial distribution, and generates a color shading correction table for adjusting the gain control of the image correction circuit to produce a shading-free image.

[0035] In some examples, the luminance correction circuitry performs luminance lens shading correction by generating luminance correction table information from shading image frames using a pre-calibrated luminance shading table based on experimental testing with one or more image capture devices. The luminance correction circuitry applies the luminance correction table information to correct for luminance lens shading.

[0036] In some examples, the apparatus includes: an image capture device that generates shadowed image frames as part of a video stream; and at least one display operatively coupled to frequency-based color correction circuitry that displays shadow-free frames. In some examples, the luminance correction circuitry, the hue unevenness removal detection circuitry, and the frequency-based color correction circuitry comprise one or more programmed processors.

[0037] According to some embodiments, a non-transitory storage medium stores executable instructions that, when executed by one or more processors, cause the one or more processors to perform luminance lens shading correction on a portion of a shadowed image frame in accordance with the methods described herein.

[0038] Figure 1 An example of apparatus 100 is shown, which is used to provide adaptive lens shading correction, for example, in at least a portion of shadow image frames from a video stream from one or more image capture devices employing lenses. For example, the video stream may be generated by a video conferencing system, a video application on a mobile device, a web-based video conferencing system, or other sources. In this example, apparatus 100 includes an image capture device 102, such as a lens system, which captures an image 103, but also introduces luminance shadows and color shadows into the captured image. The captured image 103 is provided to a frame provider 104, such as preprocessing circuitry, which performs various noise removal operations or any other suitable operations as needed. However, in other implementations, apparatus 100 does not include an image capture device or frame provider and is an intermediary device in a system such as a server in a cloud computing platform, which receives shadow image frames from various image sources, performs lens shading correction, and provides shadow-free images to other devices for display. Other variations are conceivable.

[0039] Device 100 includes adaptive lens shading correction circuitry 106, which employs frequency-based color correction (and luminance shading correction). Device 100 may include one or more displays 108 if desired, and in other implementations, device 100 provides shadow-free image frames to a network or other device for further processing, such as compression or other desired operations. In one example, adaptive lens shading correction circuitry 106 is implemented as a graphics processing unit (GPU), central processing unit (CPU), accelerated processing unit (APU), application-specific integrated circuit (ASIC), discrete logic circuitry, one or more state machines, one or more appropriately programmed processors that execute instructions stored in memory [e.g., read-only memory (ROM), random access memory (RAM), non-volatile RAM (NVRAM), or any other suitable memory]; or implemented using other suitable architectures.

[0040] In this example, the adaptive lens shading correction circuit 106 includes a brightness correction circuit 110, a tone unevenness removal and detection circuit 112, a frequency-based color correction circuit 114, and a memory storing tone distribution data 116 of the image capture device. In some implementations, the frame provider 104 accumulates frames in one or more frame buffers and provides a shadow image frame 120 to the brightness correction circuit 110, which performs brightness lens shading correction and generates a brightness-corrected shadow image frame 122 (e.g., in the form of one or more pixel macroblocks) that reaches the tone unevenness removal and detection circuit 112. The color unevenness removal detection circuit 112 removes color unevenness regions from the brightness-corrected shadow image frame 122 to identify color flat regions of the shadow image frame and provides color flat region blocks 124 (or identification information of color flat macroblocks) to a frequency-based color correction circuit. This frequency-based color correction circuit performs color shading correction on the detected color flat regions of the shadow image frame using color shading parameters including amplitude and phase parameters corresponding to the color distribution data 116 to generate a shadow-free image 126. In some implementations, the color unevenness removal detection circuit 112 receives the shadow image frame 120 and removes the unevenness regions.

[0041] In some implementations, the adaptive lens shading correction circuit 106 receives a shading image frame 120 from the frame provider 104 and uses the brightness correction circuit 110 to remove luminance shading from the shading image to produce a brightness-corrected shading image 122. A tonal unevenness removal detection circuit 112 detects macroblock-level color flat regions from the brightness-corrected shading image frame 122 and removes tonal unevenness regions to produce a color flat region block 124 with flat color areas. A frequency-based color correction circuit 114 generates a model color shading distribution map based on the image capture device tonal distribution data 116 and uses color shading distribution map parameters (e.g., amplitude and phase of a given frequency) to determine a color compensation level to adapt to the corresponding frequency found in the tonal flat region block 124. The adjusted frequency corresponds to the first harmonic frequency found in the flat region. Color shading is then removed by the frequency-based color correction circuit 114, and a shading-free image 126 is output for subsequent processing or display on one or more displays 108.

[0042] In some implementations, the adaptive lens shading correction circuit 106 provides adaptive lens shading correction in at least a portion of the shadow image frame and includes a brightness correction circuit 110 that performs brightness lens shading correction on at least a portion of the shadow image frame, a hue non-uniformity removal region detection circuit 112 that detects color flat regions in the shadow image, and a frequency-based color correction circuit 114 that generates a frequency-based color shading distribution map including color shading parameters, which include amplitude and phase parameters corresponding to the hue distribution from experimental shadow data, and generates a shadowless image by performing color shading correction on the detected color flat regions of the shadow image frame using the color shading parameters including amplitude and phase parameters corresponding to the hue distribution from experimental shadow data.

[0043] In some examples, the device includes a memory 117 that stores data representing tonal distribution, including tonal distribution data 116 for each of a plurality of color components corresponding to one or more image capture devices. In some examples, the tonal unevenness region removal detection circuit 112 detects color-flat regions in a shadow image by downsampling the shadow image frame to produce one or more downsampled pixel blocks of the shadow image frame.

[0044] In some examples, the frequency-based color correction circuit 114 uses the generated frequency-based color shading distribution map f(x,y) to produce shading model parameters:

[0045] f(x,y)=g(x)·g(y)

[0046]

[0047] Where f(x,y) is the color shading distribution, g(x) and g(y) represent the horizontal and vertical directions, and A is the amplitude. W is the phase, W is the image width, and H is the image height.

[0048] In some examples, the frequency-based color correction circuit 114 converts the hue signals detected in the color-flat regions of the pixel blocks in both the horizontal and vertical directions into the frequency domain, extracts the low-frequency components of the conversion in each direction using a low-pass filter, converts each obtained low-frequency component into a spatial distribution, and generates a color shading correction table with values ​​for adjusting the gain control of the image correction circuit to produce a shading-free image.

[0049] In some examples, the frequency-based color correction circuit 114 converts the hue in the color flat region from the spatial domain to the frequency domain, and determines the color shading correction parameters based on the color shading model parameters of the generated frequency-based color shading distribution map by minimizing the sum of hue variations.

[0050] In some examples, the brightness correction circuit 110 uses conventional techniques, employing a brightness shading table 430 based on experimental tests conducted using one or more image capture devices. Figure 4 ) Generate luminance correction table information from the shadow image frame to perform luminance lens shadow correction, and apply luminance correction table information 433 to correct luminance lens shadow.

[0051] In some examples, the apparatus includes: an image capture device that generates shadowed image frames as part of a video stream; and at least one display 108 that displays shadowless frames. In some examples, the brightness correction circuit 110, the tonal unevenness removal detection circuit 112, and the frequency-based color correction circuit 114 include one or more programmed processors. Further details regarding the various operations are provided below.

[0052] Figure 2The tonal distribution data 116 of the image capture device is illustrated graphically. The tonal distribution data 116 is generated offline using test images captured by one or more image capture devices. Graph 200 shows the one-dimensional distribution of each color component. Graph 202 shows the frequency distribution of the one-dimensional distribution of one tonal component. It has been found that, based on test images captured by one or more image capture devices, the first harmonic represents a large portion of the color shading caused by the lens. In one example, a frequency-based color correction circuit 114 stores the tonal distribution data 116 of the image capture device and generates first harmonic information for generating color shading parameters, namely amplitude and phase parameters corresponding to the tonal distribution, which is used to correct for color lens shading in the shaded image as further described below.

[0053] Figure 3 An example of a method for providing adaptive lens shading correction is shown, in which the method is performed via adaptive lens correction circuitry 106. As shown in box 300, the method begins, for example, by receiving a shadow image frame 120 as part of a video stream from frame provider 104. As shown in box 302, the method includes performing luminance lens shading correction on at least a portion of the shadow image frame, for example, via luminance correction circuitry 110. In some implementations, luminance lens shading correction is performed on a per-macroblock basis of the input shadow image frame. However, any suitable pixel basis can be used. Thus, the method includes removing luminance shading from the shadow image. In some examples, luminance shading includes preprocessing the image using any suitable techniques, such as performing luminance shading correction on each color component, demosaicing, and automatic white balance (AWB) operations, as well as other desired operations.

[0054] In some examples, based on offline experimental testing of the lens, a luminance shading table 430 for each color component was used. Figure 4The method involves determining the luminance shadows caused by the lens, which are removed using known techniques. Thus, luminance shadows are removed based on calibration information. As shown in box 304, the method includes detecting color-flat regions in the shadow image, for example, via a hue inhomogeneity region removal circuit 112. In one example, this is done on a luminance-corrected image frame. In other examples, color-flat region detection is performed on a non-luminance-corrected image. In some implementations, color-flat region detection includes downsampling the shadow image frame to produce one or more downsampled pixel blocks of the shadow image frame. As shown in box 306, the method includes generating a frequency-based color shadow distribution map including color shadow parameters, which include amplitude and phase parameters corresponding to the hue distribution from experimental shadow data based on test images captured by one or more capture devices. In one example, this is done via a frequency-based color correction circuit. For example, a frequency-based color shadow distribution map is a function representing the amount of color shadow caused by the lens on a per-color-component basis. In this example, the first harmonic is used as the frequency range corresponding to the color shadow caused by the lens.

[0055] Spectral analysis was performed to obtain the characteristics of the shading distribution map. It was determined that the shading is concentrated in the first harmonic, while shading is rare in the high-frequency band. The color shading distribution map is represented as follows:

[0056] f(x,y)=g(x)·g(y)

[0057]

[0058] Where f(x,y) is the color shading distribution, g(x) and g(y) represent the horizontal and vertical directions, W is the image width, and H is the image height.

[0059] As shown in box 308, the method includes generating a shadowless image 126 by performing color shading correction on detected color-flat regions of a shadowed image frame, for example, via a frequency-based color correction circuit, using color shading parameters comprising amplitude and phase parameters corresponding to the hue distribution from experimental shadow data. A frequency-based color shading distribution map is used to determine the amount of correction to be applied to different color levels in the scene. In one example, a color correction table is provided to the lens shadow correction circuit to correct the gain of various color components applied to various pixels in the block.

[0060] Figure 4 To show in more detail Figure 1A block diagram of an example of an adaptive lens correction circuit 106 is provided. In this example, the tone unevenness region removal detection circuit 112 includes a tone calculation circuit 400, a downsampling circuit 402, a tone gradient calculation circuit 404, and a flat region detection circuit 406. However, it will be appreciated that the other boxes shown can also be considered as part of the adaptive lens correction circuit, and that various operations of the functional boxes can be combined or separated as needed.

[0061] The tone calculation circuit 400 calculates the R / G to B / G ratio, which is considered the tone of the entire image. The input is the original image with only one color channel for each pixel. A demosaicing (interpolation) operation converts the original image into a full RGB image. Then, the tone calculation circuit 400 calculates the R / G and B / G for each interpolated pixel. The downsampling circuit 402 scales down the shadow image frame (e.g., downsamples the frame or its macroblocks), and the downsampled shadow image frame is used to determine the tone gradient, as shown in box 404. The tone gradient calculation circuit 404 uses a threshold determined through experimental operations to determine if a macroblock is a color-flat region. For example, a block is considered a color-flat region if the tone variation, brightness, and adjacent color difference are within a calibrated threshold. Other techniques for detecting color-flat regions can also be used.

[0062] Additionally, performing brightness lens shading correction via brightness correction circuit 110 includes: using brightness correction table information 432 to correct brightness shading using brightness shading table 430 based on experimental testing of one or more image capture devices. This method includes applying brightness shading table 430 to correct brightness shading using conventional techniques.

[0063] Also refer to Figures 5 to 7 The example shown illustrates color flatness region detection, where a classifier is used to eliminate areas of uneven hue. In this example, the classifier uses three indicators: brightness, color variation, and gradient information from adjacent hues. Figure 5 As shown, the color unevenness region removal detection circuit 112 classifies each M x N color block based on its brightness level, color variation 502, and gradient intensity 504, as shown in box 500, using thresholds determined experimentally to represent color flat regions. Based on the brightness, color variation, and adjacent gradient information within the color block, it determines whether the color block is in a color flat region or a color uneven (e.g., uneven) region. The flat region detection circuit 406 removes uneven regions. For example, in one example, the flat region detection circuit 406 outputs a flag indicating which macroblocks in the shadow frame are flat, as shown in data 408. The frequency-based color correction circuit 114 performs color shading correction on the color flat regions of the shadow image.

[0064] Figure 6An example of the operation of a frequency-based color correction circuit 114 is shown, in which the frequency-based color correction circuit 114 includes at least an iterative interpolation circuit 410 and a color shading extraction circuit 412. As shown in box 600, the non-uniform region is set as the average of the flat region. As shown in box 602, the frequency-based color correction circuit 114 predicts the hue values ​​of the non-uniform region and updates the hue values ​​of the non-uniform region in an iterative manner, as shown in box 604. The updated hue values ​​of the non-uniform region are then fed back to predict the regions of non-uniform hue values ​​until a particular frame is determined and thus the hue values ​​of the non-uniform region of the scene are determined. The gradient shown in 411 is used to remove the DC component by the color shading extraction operation. The color shading extraction circuit 412 generates a color shading correction table, also known as a gain table, which is used by the lens shading correction update circuit 414 (also known as an image correction circuit), which provides a block-by-block table containing appropriate gains to correct color lens shading (correcting brightness shading in some examples). The lens shading correction update circuit merges the brightness shading table 433 and the color shading table 413 into a single table in a conventional manner. In some examples, a color-corrected shading table 413 (for each color component) is provided to an image signal processor (ISP), which applies an appropriate gain and produces a shading-free image. In other examples, a lens shading correction update circuit 414 applies gain to produce a shading-free image. In some implementations, the lens shading correction update circuit includes an ISP.

[0065] For example, the color shading extraction circuit 412 converts the hue in the color flat region from the spatial domain to the frequency domain, and determines the color shading correction parameters based on the color shading model parameters of the generated frequency-based color shading distribution map, minimizing the sum of hue variations. For example, the process is shown in box 700 (…). Figure 7Starting as shown in box 702, the method includes converting the signal to the frequency domain in both the horizontal and vertical directions. For example, the conversion is: f(x,y)->g(x,v), g(u,y). f(x,y) is the spatial distribution of the signal. g(x,v) and g(u,y) are the frequency distributions in the two directions. As shown in box 704, the method includes extracting additional low-frequency components from the two conversions and removing high-frequency components to obtain low-frequency components (distributions) LFP(g(x,v)) and LFP(g(u,y)), respectively. Gradient information 411 is used to remove the DC component. As part of extracting the ultra-low frequency portion of the hue, this process removes the DC and high-frequency components from the distributions. The gradient operation is used to remove the DC component. As shown in box 706, the method includes converting two low-pass frequency distributions to spatial distributions via the following conversion: LFP(g(x,v)), LFP(g(u,y))->LFP(f(x,y)). As shown in box 708, LFP(f(x,y)) is used as the shading distribution to generate values ​​for the color shading compensation table. The corresponding frequency-based color shading distribution map parameters, namely the amplitude and phase parameters of the first harmonic frequency corresponding to the LFP(f(x,y)) frequency found in the shading image from the experimental shading data, are used to compensate for the color shading in the detected LFP(f(x,y)) distribution in the shading image.

[0066] Figure 8 This is a block diagram illustrating an exemplary method for providing adaptive lens shading correction, and shows that the generation of shading distribution maps and thresholds for color flat regions is performed offline in step 1. For example, in one example, thresholds are obtained by capturing flat-field images under different light sources for multiple sensors of the same type (e.g., sensors from the same lens). The captured raw images are averaged and then analyzed to obtain hue distribution data for the color shading distribution map (see...). Figure 2The device sets thresholds for hue variation, luminance, and adjacent color differences. In other examples, the color shading map is generated online from stored hue distribution data using this device. In some examples, the frequency-based color shading map is represented as a function f(x,y), which represents the color shading distribution caused by the image capture device. The device then applies an inverse gain to the pixels, specifically the color components, at the detected frequencies corresponding to the shading map to correct for lens-induced shading. In one example, the shading frequencies of interest are those in the first harmonic of the hue distribution obtained through experimental testing. As shown in step 2, the original image is analyzed to obtain hue-flat regions. Additionally, the following is performed: the shading image is preprocessed by luminance shading correction, interpolation, and AWB, and the image is downscaled. This operation identifies hue-flat regions if the hue variation, luminance, and adjacent color shading between pixel blocks are within calibrated thresholds. In step 3, the hue and color shading maps are transferred to the gradient domain, a transformation that allows the elimination of DC component effects. This can be done using appropriate FFT operations. In step 4, the optimal parameters for chroma shading correction are obtained by minimizing the sum of hue variations between the shading distribution map and each hue flattening block in the gradient domain. This is, for example, in... Figure 6 As shown in the image.

[0067] As further shown, the brightness correction parameter 800 is obtained experimentally and applied to the brightness shadow correction process, as illustrated by, for example, brightness shadow correction operation 802 performed by brightness correction circuit 110. As shown in box 804, the macroblocks for brightness correction of the frame are downsampled by downsampling circuit 402, and as shown in box 806, tonal flat regions of the resized blocks are used. A threshold for tonal flatness, as shown in box 810, is determined experimentally using a pre-calibration operation as shown in box 808. For example, tonal gradient calculation circuit 404 compares the detected tonal values ​​in the downsampled image with the threshold and identifies blocks or pixels within blocks that are, for example, above the tonal flatness threshold, as non-uniform or uneven regions, and these are not used as part of the color correction region of the image. As shown in box 812, for the identified tonal flat regions, frequency-based color correction circuit 114 converts those pixel blocks identified as flat regions from the frequency domain to the gradient domain. As shown in box 814, experimental information 816 is used to determine the model color shadow distribution map corresponding to the lens. In one example, the experimentally generated hue distribution data is stored in memory 216. Then, a frequency-based color correction circuit 114 generates a model color distribution map from the stored hue distribution data. A luminance shading table 430 for each component is also determined experimentally and used by the luminance shading correction operation 802. The luminance shading table 430 is also used as part of a lens shading correction circuit 414 to correct shadows on the image. As shown in box 820, the method includes converting the model color shading map from the frequency domain to the gradient domain. As shown in box 822, the method includes obtaining optimal parameters for the shading map to be placed in the shading table. For example, as previously discussed... Figure 6 As described, this involves determining the optimal correction value at a given x,y position for a given pixel.

[0068] In some implementations, device 100 includes one or more processors, such as a CPU, GPU, or other processor, that execute instructions stored in a memory such as memory 117 or other suitable memory, which, when executed, cause the processor to function as an adaptive lens shading correction circuit 106 as described herein. For example, a non-transitory storage medium includes executable instructions that, when executed by one or more processors, cause one or more processors to perform luminance lens shading correction on a portion of a shadowed image frame; detect color-flat regions in the shadowed image; generate a frequency-based color shading distribution map containing color shading parameters, including amplitude and phase parameters corresponding to the hue distribution from experimental shadow data, based on test images captured by one or more image capture devices; and generate a shadow-free image by performing color shading correction on the detected color-flat regions of the shadowed image frame using the color shading parameters, which include amplitude and phase parameters corresponding to the hue distribution from the experimental shadow data.

[0069] In some examples, the non-transitory storage medium includes executable instructions that, when executed by one or more processors, cause one or more processors to store tonal distribution data of each of a plurality of color components corresponding to one or more image capture devices, and to detect color-flat regions in a shadow image by downsampling a shadow image frame to produce one or more downsampled pixel blocks of a shadow image frame.

[0070] In some examples, the non-transitory storage medium includes executable instructions that, when executed by one or more processors, cause one or more processors to convert hues in a color-flat region from the spatial domain to the frequency domain, and to determine color shading correction parameters based on color shading model parameters of the generated frequency-based color shading distribution map and to minimize the sum of hue variations.

[0071] In some examples, the non-transitory storage medium includes executable instructions that, when executed by one or more processors, cause one or more processors to generate a frequency-based color shading distribution map f(x,y), which is represented as:

[0072] f(x,y)=g(x)·g(y)

[0073]

[0074] Where f(x,y) is the color shading distribution, g(x) and g(y) represent the horizontal and vertical directions, and A is the amplitude. W is the phase, W is the image width, and H is the image height.

[0075] In addition, in some examples, the non-transitory storage medium includes executable instructions that, when executed by one or more processors, cause one or more processors to perform luminance lens shading correction by generating luminance correction table information from a shading image frame using a pre-calibrated luminance shading table based on experimental testing of one or more image capture devices and by applying the luminance correction table information to correct luminance shading.

[0076] Among other technical advantages, unlike other techniques that use average shading, frequency-based color correction techniques use shading models derived from the frequency characteristics of chroma shading. The frequency characteristics of the chroma shading processing described herein provide more accurate chroma shading correction, resulting in higher quality video images. Furthermore, the disclosed methods and apparatus can employ low computational load and high execution efficiency, as in some examples using downsampled low-resolution pixel information. In some implementations, downsampled pixel block information leads to lower memory consumption compared to other adaptive techniques.

[0077] Although the features and elements have been described above in specific combinations, each feature or element may be used alone without other features and elements, or in various combinations with or without other features and elements. In some implementations, the apparatus described herein is manufactured using a computer program, software, or firmware incorporated into a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of computer-readable storage media include read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor storage devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital universal disks (DVDs).

[0078] In the foregoing detailed description of various embodiments, reference has been made to the accompanying drawings, which form a part thereof, and specific preferred embodiments in which the invention can be practiced are illustrated by way of example. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the invention, and it should be understood that other embodiments may be utilized, and logical, mechanical, and electrical changes may be made without departing from the scope of the invention. To avoid details unnecessary for enabling those skilled in the art to practice the invention, certain information known to those skilled in the art may have been omitted in the description. Furthermore, those skilled in the art can readily construct many other variations of the embodiments incorporating the teachings of this disclosure. Therefore, the invention is not intended to be limited to the specific forms described herein, but rather is intended to cover such alternatives, modifications, and equivalents that may reasonably be included within the scope of the invention. Therefore, the foregoing detailed description is not restrictive, and the scope of the invention is defined only by the appended claims. The foregoing detailed description of the embodiments and examples described herein has been given for purposes of illustration and description only and not as a limitation. For example, the described operations may be performed in any suitable order or manner. Therefore, it is contemplated that the invention covers any and all modifications, variations, or equivalents falling within the scope of the foregoing disclosure and the basic principles claimed herein.

[0079] The detailed descriptions above and the examples described herein are given for illustrative and descriptive purposes only and are not intended to be restrictive.

Claims

1. A method for providing adaptive lens shading correction in at least a portion of a shadowed image frame, the method comprising: Perform luminance lens shading correction on a portion of the shadow image frame; Detect color-flat regions in the shadow image frame; Generate a frequency-based color shading distribution map including color shading parameters, the color shading parameters including amplitude and phase parameters corresponding to the hue distribution of experimental shading data based on test images captured by one or more image capture devices; as well as A shadowless image is generated by performing color shadow correction on the detected color-flat regions of the shadowed image frame using the color shadow parameters.

2. The method according to claim 1, wherein: The tone distribution consists of stored tone distribution data corresponding to each of a plurality of color components of the one or more image capture devices; as well as Detecting the color-flat region in the shadow image includes downsampling the shadow image frame to generate one or more downsampled pixel blocks of the shadow image frame.

3. The method according to claim 2, comprising: The hue in the color flat region is converted from the spatial domain to the frequency domain; and color shading correction parameters are determined based on the color shading parameters of the generated frequency-based color shading distribution map. And to minimize the total variation in hue.

4. The method according to claim 1, wherein, The generated frequency-based color shading distribution map f(x,y) is represented as: f(x,y)=g(x)·g(y) Where f(x,y) is the color shading distribution, g(x) and g(y) represent the horizontal and vertical directions, and A is the amplitude. W is the phase, W is the image width, and H is the image height.

5. The method according to claim 3, wherein, Converting the hue in the color flat area includes: The hue signals detected in the color-flat regions of the pixel block are converted into the frequency domain in both the horizontal and vertical directions; Use a low-pass filter to extract the low-frequency component of each direction transition; Each obtained low-frequency component is converted into a spatial distribution; and determining the color shading correction parameters includes generating a color shading correction table for adjusting the gain control of the image correction circuit to produce a shading-free image.

6. The method according to claim 1, wherein, Performing luminance lens shading correction includes generating luminance correction table information from the shading image frame using a pre-calibrated luminance shading table based on experimental testing of one or more image capture devices, and applying the luminance correction table information to correct the luminance shading.

7. An apparatus for providing adaptive lens shading correction in at least a portion of a shadowed image frame, the apparatus comprising: A brightness correction circuit that is operable to perform brightness lens shading correction on at least a portion of the shadow image frame; A color unevenness removal detection circuit is operable to detect color flat areas in the shadow image frame; Frequency-based color correction circuits can operate as follows: Generate a frequency-based color shading distribution map including color shading parameters, the color shading parameters including amplitude and phase parameters corresponding to the hue distribution of experimental shading data based on test images captured by one or more image capture devices; as well as A shadowless image is generated by performing color shadow correction on the detected color-flat regions of the shadowed image frame using the color shadow parameters.

8. The apparatus of claim 7, further comprising a memory storing data representing the hue distribution, the data including hue distribution data for each of a plurality of color components corresponding to the one or more image capture devices.

9. The apparatus according to claim 8, wherein, The color unevenness removal detection circuit is operable to detect the color flat regions in the shadow image by downsampling the shadow image frame to generate one or more downsampled pixel blocks of the shadow image frame.

10. The apparatus according to claim 9, wherein, The frequency-based color correction circuit is operable to convert hues in color flat regions from the spatial domain to the frequency domain, and to determine color shading correction parameters by minimizing the sum of hue variations based on the color shading parameters of the generated frequency-based color shading distribution map.

11. The apparatus according to claim 10, wherein, The frequency-based color correction circuit is operable to generate the color shading parameters using the generated frequency-based color shading distribution map f(x,y) in the following form: f(x,y)=g(x)·g(y) Where f(x,y) is the color shading distribution, g(x) and g(y) represent the horizontal and vertical directions, and A is the amplitude. W is the phase, W is the image width, and H is the image height.

12. The apparatus according to claim 10, wherein, The frequency-based color correction circuit can operate as follows: The hue signals detected in the color-flat regions of the pixel block are converted into the frequency domain in both the horizontal and vertical directions; Use a low-pass filter to extract the low-frequency component of each direction transition; Each obtained low-frequency component is converted into a spatial distribution; and a color shading correction table is generated to adjust the gain control of the image correction circuit to produce the shading-free image.

13. The apparatus according to claim 7, wherein, The brightness correction circuit is operable to perform brightness lens shading correction by generating brightness correction table information from the shading image frame using a pre-calibrated brightness shading table based on experimental testing of one or more image capture devices and by applying the brightness correction table information to correct brightness lens shading.

14. The apparatus of claim 11, comprising: An image capture device, operable to generate the shadow image frames as part of a video stream, and At least one display is operatively coupled to the frequency-based color correction circuit and is operable to display the shadowless image.

15. The apparatus according to claim 7, wherein, The brightness correction circuit, the hue unevenness removal detection circuit, and the frequency-based color correction circuit are composed of one or more programmed processors.

Images