System and method for implementing flicker cancellation without using flicker independent frames

Abstract

A flicker elimination system includes a compensation gain estimation circuit that receives a plurality of flicker-related frames having a sliding band, and estimates a compensation gain for each flicker-related frame from the flicker-related frames, and a flicker elimination circuit that eliminates flicker of each flicker-related frame. The flicker cancellation circuit applies flicker compensation to each flicker-related frame according to the compensation gain.

Description

Technical Field

[0001] The present invention relates to post - processing of frames provided by an image source, and more particularly, to a de - flickering system and related method for compensating gain estimation using flicker - related frames with a sliding band. Background Art

[0002] Rolling shutters are commonly used in complementary metal - oxide - semiconductor (CMOS) image sensors. Different from global shutter cameras that acquire all pixels of a frame simultaneously, rolling shutter cameras acquire pixels of a frame in a row - by - row manner. However, this may cause problems under alternating current (AC) light sources powered by an AC voltage signal. If the shutter time is not synchronized with the light source frequency (for example, the shutter time is not on the flicker basis, that is, not on the shortest period of the alternating current in the environment), the time lag between exposures of each row (pixel row) may result in obvious bright and dark bands, namely the so - called flicker. A typical solution is to make the shutter time not shorter than the flicker basis, which leads to a reduced dynamic range in scenes with bright illumination, artifacts in fast - moving objects, and over - exposure of bright points. Therefore, an innovative de - flickering framework is needed that can eliminate / mitigate flicker without using any reference frames (for example, flicker - independent frames that are free of flicker due to acquisition under the condition that the shutter time of the rolling shutter is an integer multiple of the flicker basis). Summary of the Invention

[0003] One object of the present invention is to provide a de - flickering system and its related method for compensating gain estimation using flicker - related frames with a sliding band.

[0004] According to a first aspect of the present invention, an exemplary de - flickering system is proposed. The exemplary de - flickering system includes a compensation gain estimation circuit and a flicker elimination circuit. The compensation gain estimation circuit is configured to receive a plurality of flicker - related frames with a sliding strip, and estimate the compensation gain of each flicker - related frame according to the plurality of flicker - related frames. The flicker elimination circuit is configured to apply flicker compensation to each flicker - related frame among the plurality of flicker - related frames according to the compensation gain.

[0005] According to a second aspect of the present invention, an exemplary de - flickering method is disclosed. The exemplary de - flickering method includes: receiving a plurality of flicker - related frames with a sliding strip; estimating the compensation gain of each flicker - related frame according to the plurality of flicker - related frames; and applying flicker compensation to each flicker - related frame among the plurality of flicker - related frames according to the compensation gain.

[0006] After reading the following detailed description of the preferred embodiments shown in the various figures, these and other objects of the present invention will no doubt become apparent to those of ordinary skill in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To gain a more comprehensive understanding of the present invention, read the following detailed description and examples and refer to the accompanying drawings, in which:

[0008] Figure 1 Schematic diagram of a de - flickering system showing an embodiment of the present invention.

[0009] Figure 2 Schematic diagram showing the de - flickering process of the flicker - related frames obtained in the first camera mode provided by an embodiment of the present invention.

[0010] Figure 3 Diagram showing the application of the de - flickering process to the flicker - related frames obtained in the second camera mode according to an embodiment of the present invention.

[0011] Figure 4 Schematic diagram showing the de - flickering process of the flicker - related frames obtained in the third camera mode provided by an embodiment of the present invention.

[0012] Figure 5 Diagram showing the estimation of the sine wave form most suitable for the residual signal according to an embodiment of the present invention.

[0013] Figure 6 Diagram showing the estimation of the flicker signal that causes a sliding band between two flicker - related frames according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0014] Some terms are used in the following description and claims that refer to specific components. As those skilled in the art will understand, electronic device manufacturers may use different names to refer to a certain component. It is not the intention herein to distinguish components that have different names but the same function. In the following description and claims, the terms "comprising" and "including" are used in an open - ended manner and should thus be interpreted as "including but not limited to...". In addition, the term "coupled" is intended to mean an indirect or direct electrical connection. Thus, if a device is coupled to another device, the connection may be by a direct electrical connection or by an indirect electrical connection via other devices and connections.

[0015] Figure 1Schematic diagram of the anti-flicker system according to an embodiment of the present invention. As an example, but not limited thereto, the anti-flicker system 100 can be used by a mobile device such as a mobile phone. The anti-flicker system 100 includes a compensation gain estimation circuit 102 and a flicker elimination circuit 104. The compensation gain estimation circuit 102 and the flicker elimination circuit 104 can be implemented in the same chip or different chips, depending on the actual design considerations. The compensation gain estimation circuit 102 is configured to receive a plurality of flicker-related frames (i.e., non-flicker-free acquired frames) F_1 - F_N (N≥2) having a sliding band, and estimate the compensation gains CG_1 - CG_N of the flicker-related frames F_1 - F_N based on the flicker-related frames F_1 - F_N. For example, the flicker-related frames (i.e., non-flicker-free frames) F_1 - F_N are captured by a rolling shutter camera under the condition that the scene to be captured is illuminated by one or more alternating current light sources (which are flicker sources and may cause periodic flicker artifacts in the captured frames). As Figure 1 shown, the flicker-related frame F_1 includes a plurality of alternating dark bands BD and a plurality of bright bands BB, where each band may include one or more pixel rows. Similarly, the flicker-related frame F_N includes a plurality of alternating dark bands BD and a plurality of bright bands BB, where each band may include one or more pixel rows. The frames F_1 to F_N are not limited to a static environment. It should be noted that the positions of the dark bands BD in the flicker-related frame F_N are different from the positions of the dark bands BD in the flicker-related frame F_1, and the positions of the bright bands BB in the flicker-related frame F_N are also different from the positions of the bright bands BB in the flicker-related frame F_1. Therefore, compared with the dark bands BD in the flicker-related frame F_1 (which may be acquired at an earlier time point), the dark bands BD in the flicker-related frame F_N (which may be acquired at a later time point) slide, and compared with the bright bands BB in the flicker-related frame F_1 (which may be acquired at an earlier time point), the bright bands BB in the flicker-related frame F_N (which may be acquired at a later time point) slide. The compensation gain estimation circuit 102 can utilize the characteristic that two or more flicker-related frames F_1 - F_N have sliding bands to obtain the compensation gains CG_1 - CG_N. The flicker elimination circuit 104 is used to perform flicker compensation on the flicker-related frames F_1 - F_N respectively according to the compensation gains CG_1 - CG_N, and output the flicker-compensated frames F_1' - F_N'. Specifically, the compensation gain CG_1 is used to eliminate / reduce the flicker (i.e., the alternation of bright bands and dark bands) of the flicker-related frame F_1 caused by the alternating current light source, and the compensation gain CG_N is used to eliminate / reduce the flicker (i.e., the alternation of bright bands and dark bands) of the flicker-related frame F_N caused by the alternating current light source.

[0016] In some embodiments of the present invention, the anti-flicker system 100 and the camera module can be co-located on the same mobile device (e.g., a mobile phone), and the anti-flicker system 100 can obtain the flicker-related frames F_1 - F_N required for compensation gain estimation from the camera module.

[0017] Figure 2 This is a schematic diagram of the de - flickering process for the flicker - related frames obtained in the first camera mode in an embodiment of the present invention. In this embodiment, the compensation gain estimation circuit 102 receives the flicker - related frames F_1 - F_N (N≥2) from the camera module 200 operating in the conventional single - exposure mode. It should be noted that the shutter time is not required to be based on the flicker reference. For example, the shutter time is not an integer multiple of the flicker reference and can be shorter than the flicker reference.

[0018] Figure 3 This is a diagram showing the de - flickering process applied to the flicker - related frames obtained in the second camera mode according to an embodiment of the present invention. In this embodiment, the compensation gain estimation circuit 102 receives the flicker - related frames F_1 - F_N (N≥2) from the camera module 300 operating in the dual - gain high dynamic range (DG - HDR) mode. Therefore, different gains (e.g., analog gain or digital gain) are applied to generate two frames from a single image capture. For example, the DG - HDR mode can be a dual - conversion gain (DCG) mode with a high conversion gain (HCG) and a low conversion gain (LCG). The HCG is used to increase the conversion gain of the signal and is usually used under low - light conditions to improve the signal - to - noise ratio. The LCG is used to reduce the conversion gain of the signal and is usually used under high - light conditions to avoid signal saturation. Therefore, a sequence of flicker - related frames F_1 - F_N ( Figure 3 labeled as F_L1 - F_LN in Figure 3Marked as F_H1 - F_HN in the figure), where the flicker - related frame F_Li (LCG mode) and the flicker - related frame F_Hi (HCG mode) are two frames generated by applying different conversion gains to a single captured frame, where i = {1, 2, …, N}. For another example, the DG - HDR mode can be a dual - analog - gain (DAG) mode with high analog gain and low analog gain. The high analog gain is used to amplify the sensor output signal, usually used under low - light conditions to increase the signal strength, and the low analog gain is used to reduce the amplification of the sensor output signal, usually used under high - light conditions to avoid signal saturation. Therefore, a sequence of flicker - related frames F_L1 - F_LN is generated in a low - analog - gain mode, and another sequence of flicker - related frames F_H1 - F_HN is generated in a high - analog - gain mode. The flicker - related frame F_Li (one analog - gain mode) and the flicker - related frame F_Hi (another analog - gain mode) are two frames generated by applying different analog gains to a single captured frame, where i = {1, 2, …, N}. It should be noted that the shutter time is not required to be based on the flicker reference. For example, the shutter time is not an integer multiple of the flicker reference and can be shorter than the flicker reference. In this embodiment, the high and low gains usually depend on the photosensitive element, the design of the camera, the shooting scene, the lighting conditions, the image quality, etc. In this embodiment, generating flicker - related frames using different analog - gain modes (high analog gain and low analog gain) and different conversion - gain modes (high conversion gain and low conversion gain), and the shutter time not being required to be based on the flicker reference, this design has the following advantages: improving the dynamic range. By applying different conversion gains and analog gains during the same frame acquisition process, more luminance information can be obtained. Using high conversion gain and high analog gain under low - light conditions can obtain more details and reduce noise; using low conversion gain and low analog gain under high - light conditions can avoid signal saturation and retain more luminance details. This dual - gain mode (DCG and DAG) or only using one of them can effectively expand the dynamic range of the image, enabling the capture of details in both dark and bright parts in high - contrast scenes. The shutter time not being required to be based on the flicker reference means that the shutter time can be flexibly adjusted according to specific shooting requirements: a short shutter time can capture fast - moving objects and reduce motion blur, and a long shutter time can increase the exposure time under low - light conditions to obtain more light. By using different high - and - low conversion gains and / or analog gains to generate flicker - related frames and flexibly setting the shutter time, the dynamic range of the image can be significantly improved, the flicker effect can be reduced, the image quality can be improved, and it can adapt to various shooting scenes and conditions. This design enables the camera to achieve better imaging effects under various complex lighting and motion conditions.

[0019] The compensation gain estimation circuit 102 can obtain compensation gains CG_L1 - CG_LN and CG_H1 - CG_HN by utilizing the characteristic that two or more flicker - related frames F_L1 - F_LN and F_H1 - F_HN have a sliding stripe. The compensation gains CG_L1 - CG_LN and CG_H1 - CG_HN can be the same gains respectively or different gains. The flicker elimination circuit 104 is used to perform flicker compensation on the flicker - related frames F_L1 - F_LN and F_H1 - F_HN respectively according to the compensation gains CG_L1 - CG_LN and CG_H1 - CG_HN, and output the corresponding frames F_Li’ and F_Hi‘ (i = 1…N) after flicker compensation respectively. In this embodiment Figure 3 takes the input of F_L1 - F_LN and F_H1 - F_HN to generate the corresponding frames F_L1’ and F_H1’ after flicker compensation as an example. Those of ordinary skill in the art can know that Figure 3 it is also possible to input F_L1 - F_LN and F_H1 - F_HN to generate the corresponding frames F_L2’ and F_H2’ after flicker compensation respectively, or F_L2’ and F_H2’… or F_LN’ and F_HN’.

[0020] Figure 4 FIG. shows a diagram of the de - flicker process applied to flicker - related frames obtained in the third camera mode according to an embodiment of the present invention. In this embodiment, the compensation gain estimation circuit 102 receives the flicker - related frames F_1 and F_N (N = 2) from a camera module 400 operating in any other HDR mode and having no flicker - independent frames (i.e., non - flickering frames). For example, the flicker - related frames F_1 and F_N (N = 2) can be sequentially captured by the camera module 400 under different exposure settings, where the shutter time can be shorter than the flicker reference.

[0021] The compensation gain estimation circuit 102 is used to estimate the compensation gains CG_1 - CG_N of the flicker - related frames F_1 - F_N respectively. In the first compensation gain estimation design, the compensation gain estimation circuit 102 can use a mathematical model (e.g., a periodic function) to model each compensation gain and estimate the parameters of the mathematical model according to the flicker - related frames F_1 - F_N with a sliding band. Each compensation gain CG_1 - CG_N depends on the flicker signal affecting the corresponding flicker - related frame. The flicker signal can be modeled by a sine waveform or a non - sine waveform (e.g., a triangular wave or a square wave), depending on the design considerations. To make the technical features of the present invention easier to understand, it is assumed hereinafter that the flicker signal is modeled by a sine waveform A·sin(θ), and the compensation gain estimation circuit 102 commonly uses two flicker - related frames F_1 and F_N (N = 2) for compensation gain estimation.

[0022] Please refer to Figure 5 and Figure 6 ,Figure 5 Schematic diagram of estimating the sine waveform of the most suitable residual signal according to an embodiment of the present invention . Figure 6 Schematic diagram of estimating the flicker signals Asin(α) and Asin(β) that cause the sliding stripe between two flicker-related frames F_1 and F_2 according to an embodiment of the present invention. As Figure 5 shown, the compensation gain estimation circuit 102 obtains a feature signal S1 from the flicker-related frame F_1 and another feature signal S2 from the flicker-related frame F_2. For example, the feature signal S1 can be derived from a partial frame 502 selected from the flicker-related frame F_1, and the feature signal S2 can be derived from a partial frame 504 selected from the flicker-related frame F_2, where the selection of the partial frames 502 and 504 can be carried out with / without external segmentation information. The feature signal S1 can be obtained by averaging the pixel values of the pixels on each row of the partial frame 502. The feature signal S2 can be obtained by averaging the pixel values of the pixels on each row of the partial frame 504. The residual signal R can be obtained by dividing one of the feature signals S1 and S2 by the other, or can be obtained by subtracting one of the feature signals S1 and S2 from the other. In this embodiment, the residual signal R can be the difference between the feature signals S1 and S2. After obtaining the residual signal R, the compensation gain estimation circuit 102 estimates the sine waveform parameters (such as amplitude D and phase )(such as R = S2 - S1), where for example, the compensation gain estimation circuit 102 can adopt Fourier transform or other fitting algorithms. The phase difference comes from the setting of the camera. In φ = 2π·Frac(f·Δt), f is the frequency of the AC, and Δt is the difference between two sliding frames. The frequency information can be provided externally or defined by the user. After successfully estimating the amplitude D and phase of the sine waveform of the residual signal R, the compensation gain estimation circuit 102 finds A, α, and β that can satisfy the following formula.

[0023]

[0024] The compensation gain CG_1 of the flicker-related frame F_1 can be determined based on the flicker signal Asin(α). Specifically, the flicker signal Asin(α) can be used to determine the compensation values of the compensation gain CG_1 for flicker compensation of different rows of the flicker-related frame F_1. After applying each compensation value to (e.g., adding or multiplying) the pixel values of at least a part (e.g., part or all) of the pixels in the same row selected from the rows of the flicker-related frame F_1, a flicker-free or flicker-reduced flicker compensation frame F_1' can be generated. Similarly, the compensation gain CG_2 of the flicker-related frame F_2 can be determined based on the flicker signal Asin(β). Specifically, the flicker signal Asin(β) can be utilized to determine the compensation values of the compensation gain CG_2 for flicker compensation of different rows of the flicker-related frame F_2. After applying each compensation value to (e.g., adding or multiplying) the pixel values of at least a part (e.g., part or all) of the pixels in the same row selected from each row of the flicker-related frame F_2, a flicker-free or flicker-reduced flicker compensation frame F_2' can be generated.

[0025] It should be noted that the calculation of the compensation gain is not limited to the above method based on the mathematical model. In fact, any method that can utilize two or more flicker-related frames with sliding bands to determine the compensation gain can be adopted by the compensation gain estimation circuit 102.

[0026] In some embodiments of the present invention, the compensation gain estimation circuit 102 can adopt an artificial intelligence (AI) assisted method for compensation gain estimation. For example, the compensation gain estimation circuit 102 can utilize a neural network to estimate the compensation gains CG_1 - CG_N, where the neural network can take the flicker-related frames F_1 - F_N as its inputs.

[0027] In some embodiments of the present invention, the compensation gain estimation circuit 102 can respectively derive multiple reduced frames from the flicker-related frames F_1 - F_N, and estimate the compensation gains CG_1 - CG_N of the flicker-related frames F_1 - F_N based on the reduced frames. For example, the method based on the mathematical model can obtain a feature signal from some frames selected from the reduced frames (generated by reducing the flicker-related frames F_1 - F_N). For another example, the AI assisted method can use the reduced frames (generated by reducing the flicker-related frames F_1 - F_N) as the inputs of the neural network.

[0028] As described above, the flicker cancellation circuit 104 applies flicker compensation to the flicker-related frames F_1 - F_N according to the compensation gains CG_1 - CG_N, respectively. In the first flicker cancellation design, the flicker cancellation circuit 104 can apply flicker compensation to each row included in each flicker-related frame F_1 - F_N. For example, the flicker compensation can be applied to each entire row. As another example, the flicker compensation can be local flicker compensation that is only applied to a part of each row. In the second flicker cancellation design, the flicker cancellation circuit 104 can apply flicker compensation only to a part of the rows included in each flicker-related frame F_1 - F_N. Thus, only some rows in a flicker-related frame will undergo flicker compensation. For example, the flicker cancellation circuit 104 can apply flicker compensation to the brighter rows included in the bright band using one compensation gain setting. For example, the flicker cancellation circuit 104 can apply flicker compensation to the darker rows included in the dark band using another compensation gain setting. In the third flicker cancellation design, the flicker cancellation circuit 104 can utilize a neural network to control the flicker compensation applied to each flicker-related frame F_1 - F_N. For example, artificial intelligence can be used to determine the number of selected rows in a flicker-related frame that require flicker compensation and / or the magnitude of the flicker compensation to be applied to a selected row.

[0029] Compared with traditional de-flickering methods, the proposed de-flickering method provides the following key improvements. The de-flickering process performed by the de-flickering system 100 has low computational complexity and can be executed on a mobile phone, thus having portability. There is no need for a flicker-independent frame (i.e., a non-flickering frame) as a reference. Therefore, the de-flickering system 100 can work in cooperation with a camera module operating in a conventional single-exposure mode, a DG-HDR mode, or any other HDR mode. Thus, a camera module without a specific shutter time limit can increase the dynamic range of DG-HDR images, especially in scenes illuminated by bright light, be able to capture fast-moving objects that require a shorter shutter time, enhance the overall image quality to provide a more attractive visual experience, and / or facilitate advanced image processing tasks such as segmentation.

[0030] Those skilled in the art will readily observe that many modifications and changes can be made to the apparatus and method while retaining the teachings of the present invention. Therefore, the above disclosure should be understood to be limited only by the bounds and scope of the appended claims.

Claims

1. A de - flickering system, comprising: A compensation gain estimation circuit configured to receive a plurality of flicker - related frames having a sliding strip, and estimate a compensation gain for each flicker - related frame based on the plurality of flicker - related frames; A flicker cancellation circuit configured to apply flicker compensation to each flicker - related frame among the plurality of flicker - related frames based on the compensation gain.

2. The anti-flicker system according to claim 1, wherein Wherein, The compensation gain estimation circuit uses a mathematical model to model the compensation gain, and estimates the parameters of the mathematical model based on the plurality of flicker - related frames.

3. The anti-flicker system according to claim 1, characterized in that, The compensation gain estimation circuit uses a neural network to estimate the compensation gain.

4. The anti-flicker system according to claim 1, wherein The compensation gain estimation circuit respectively derives a plurality of reduced frames from the plurality of flicker - related frames, and estimates the compensation gain for each flicker - related frame among the plurality of flicker - related frames based on the plurality of reduced frames.

5. The anti-flicker system according to claim 1, characterized in that, The flicker cancellation circuit applies the flicker compensation to each row included in each flicker - related frame among the plurality of flicker - related frames.

6. The de - flickering system according to claim 1, characterized in that, The flicker cancellation circuit applies the flicker compensation only to a part of the rows included in each flicker - related frame among the plurality of flicker - related frames.

7. The anti-flicker system according to claim 1, characterized in that The flicker cancellation circuit uses a neural network to control the flicker compensation applied to each flicker - related frame among the plurality of flicker - related frames.

8. The anti-flicker system according to claim 1, wherein The compensation gain estimation circuit receives the plurality of flicker - related frames from a camera module operating in a conventional single - exposure mode.

9. The de-flickering system according to claim 1, wherein The compensation gain estimation circuit receives the plurality of flicker - related frames from a camera module operating in a dual - gain high - dynamic - range (DG - HDR) mode.

10. The anti-flicker system according to claim 1, characterized in that, The compensation gain estimation circuit receives the plurality of flicker - related frames from a camera module operating in a high - dynamic - range (HDR) mode without flicker - independent frames.

11. A de - flickering method, comprising: Receiving a plurality of flicker - related frames having a sliding strip; Estimating a compensation gain for each flicker - related frame among the plurality of flicker - related frames based on the plurality of flicker - related frames; And Applying flicker compensation to each flicker - related frame among the plurality of flicker - related frames based on the compensation gain.

12. The anti-flicker method according to claim 11, wherein Estimating a compensation gain for each flicker - related frame among the plurality of flicker - related frames based on the plurality of flicker - related frames includes: Using a mathematical model to model the compensation gain; and Estimating the parameters of the mathematical model based on the plurality of flicker - related frames.

13. The de - flashing method according to claim 11, characterized in that, Estimating a compensation gain for each flicker - related frame among the plurality of flicker - related frames based on the plurality of flicker - related frames includes: Using a neural network to estimate the compensation gain.

14. The de-flickering method according to claim 11, wherein Estimating a compensation gain for each flicker - related frame among the plurality of flicker - related frames based on the plurality of flicker - related frames includes: Respectively deriving a plurality of reduced frames from the plurality of flicker - related frames; and Estimating the compensation gain for each flicker - related frame among the plurality of flicker - related frames based on the plurality of reduced frames.

15. The de - flashing method according to claim 11, characterized in that, Applying flicker compensation to each flicker - related frame among the plurality of flicker - related frames based on the compensation gain includes: Applying the flicker compensation to each row included in each flicker - related frame among the plurality of flicker - related frames.

16. The de - flashing method according to claim 11, wherein Apply the flicker compensation to each of the plurality of flicker-related frames according to the compensation gain, and apply the flicker compensation only to a part of the lines included in each of the plurality of flicker-related frames.

17. The de - flashing method according to claim 11, wherein Applying the flicker compensation to each of the plurality of flicker-related frames according to the compensation gain includes: Controlling, by using a neural network, the flicker compensation applied to each of the plurality of flicker-related frames.

18. The de-flickering method according to claim 11, wherein Receiving the plurality of flicker-related frames having a sliding stripe includes: Receiving the plurality of flicker-related frames output from a camera module operating in a conventional single-exposure mode.

19. The de - flashing method according to claim 11, wherein, Receiving the plurality of flicker-related frames having a sliding stripe includes: Receiving the plurality of flicker-related frames output from a camera module operating in a dual-gain high dynamic range (DG-HDR) mode.

20. The anti-flickering method according to claim 11, characterized in that, Receiving the plurality of flicker-related frames having a sliding stripe includes: Receiving the plurality of flicker-related frames output from a camera module operating in a high dynamic range (HDR) mode without flicker-independent frames.

Images