Imaging device for generating high dynamic range image

Abstract

The present invention relates to an imaging device for generating a high dynamic range image. One embodiment of such present invention may comprise: an image capture unit for capturing a plurality of images having different exposure values from each other, and generating a long exposure image, a short exposure image, a first extreme exposure image, and a second extreme exposure image; a first synthesis unit for generating an image for LED flicker mitigation by synthesizing the long exposure image, the short exposure image, and the first extreme exposure image; a second synthesis unit for generating a motion image by synthesizing the long exposure image, the short exposure image, and the second extreme exposure image; and a blending unit for generating a synthesized image by synthesizing the image for LED flicker mitigation and the motion image on the basis of a motion weight and a disparity value indicating the difference in exposure ratio of each location in an image.

Description

Imaging device that produces wide dynamic range images

[0001] The present invention relates to an imaging device for generating a wide dynamic range image.

[0002] The real world, as perceived by humans, can have varying levels of brightness. There can be significant differences in brightness between bright areas illuminated by light and dark areas in shadows. Conventional imaging devices have limitations in accurately capturing images with varying levels of brightness.

[0003] HDR (high dynamic range) imaging devices can address these issues by supporting a wide dynamic range. HDR imaging devices can create HDR images by combining multiple images with different exposures.

[0004] An embodiment of the present invention provides an imaging device capable of improving the accuracy of synthesis of an image for mitigating LED (Light emitting diode) flicker and a motion image in an HDR imaging device supporting LFM (Led Flicker Mitigation) technology.

[0005] An imaging device for generating a wide dynamic range image according to an embodiment of the present invention may include an image capture unit that captures a plurality of images having different exposure values ​​to generate a long exposure image, a short exposure image, a first extreme exposure image, and a second extreme exposure image; a first synthesis unit that synthesizes the long exposure image, the short exposure image, and the first extreme exposure image to generate an LED flicker mitigation image; a second synthesis unit that synthesizes the long exposure image, the short exposure image, and the second extreme exposure image to generate a motion image; and a blending unit that synthesizes the LED flicker mitigation image and the motion image based on a disparity value and a motion weight indicating a difference in exposure ratio according to a position of the image to generate a composite image.

[0006] An embodiment of the present invention provides an effect of securing a wider dynamic range in an HDR imaging device supporting LFM (Led Flicker Mitigation) technology.

[0007] FIG. 1 is a block diagram of an imaging device according to an embodiment of the present invention.

[0008] Figure 2 is a detailed configuration diagram of the image capture unit of Figure 1.

[0009] Fig. 3 is a configuration diagram of the pixel array of Fig. 2.

[0010] Fig. 4 is a drawing for explaining the operation of the weight application unit of Fig. 1.

[0011] Figure 5 is a detailed configuration diagram of the blending section of Figure 1.

[0012] Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components will be given the same reference numerals, even if they appear in different drawings. Furthermore, when describing embodiments of the present invention, detailed descriptions of related known structures or functions will be omitted if they are deemed to hinder understanding of the embodiments of the present invention.

[0013] FIG. 1 is a block diagram of an imaging device according to an embodiment of the present invention.

[0014] Referring to FIG. 1, the imaging device (100) may include an image capture unit (105), a first synthesis unit (110), a second synthesis unit (115), a disparity estimation unit (120), a weight application unit (125), a blending unit (130), and / or a tone mapping application unit (135).

[0015] The imaging device (100) can capture multiple images with different exposure values ​​to generate an image capable of expressing a wider dynamic range, and synthesize the multiple images to generate an image capable of expressing a wider dynamic range. The imaging device (100) can generate an image capable of expressing a wider dynamic range as the exposure ratio (e.g., the difference between exposure values ​​in the multiple images) in the captured multiple images increases.

[0016] The image capture unit (105) can capture multiple images having different exposure values ​​to generate a long exposure image (L), a short exposure image (S), and extreme exposure images (VS1, VS2). In FIG. 1, the image capture unit (105) is illustrated as capturing three images, namely a long exposure image, a short exposure image, and an extreme exposure image, but the embodiments of the present disclosure are not limited thereto.

[0017] For example, the image capture unit (105) can adjust the exposure time to have different exposure values ​​among multiple images. The exposure time may refer to the length of time that the image capture unit (105) is exposed to light. “Exposure time” may be referred to as a term referring to the length of time that the image capture unit (105) receives light to capture an image. The image capture unit (105) may include a configuration (e.g., an electronic shutter) for adjusting the exposure time.

[0018] Additionally, the image capture unit (105) can profile noise to generate a noise profile value (NP). Here, the noise profile may refer to a noise level value that varies depending on changes in the data characteristics of the image. The detailed configuration and operation of the image capture unit (105) will be described in more detail in FIGS. 2 and 3 described below.

[0019] The first synthesis unit (110) can generate an LED flicker mitigation (Led Flicker Mitigation, hereinafter referred to as 'LFM') image (LFMI) by synthesizing a long exposure image (L), a short exposure image (S), and an extreme exposure image (VS1). For example, the first synthesis unit (110) can generate an LFM image (LFMI) by synthesizing a long exposure image (L), a short exposure image (S) captured from a large pixel (described later), and an extreme exposure image (VS1) captured from a low-sensitivity small pixel (described later). The imaging device (100) can prevent or alleviate an LED flicker phenomenon that may occur when photographing an LED light source by the LED flicker mitigation image (LFMI).

[0020] The second synthesis unit (115) can generate a motion image (MOTI) by synthesizing a long exposure image (L), a short exposure image (S), and an extreme exposure image (VS2). For example, the second synthesis unit (115) can generate a motion image (MOTI) by synthesizing a long exposure image (L), a short exposure image (S) captured from a large pixel, and an extreme exposure image (VS2) captured from a large pixel of normal sensitivity (described later). Here, motion can represent the degree of movement of an image. In addition, the motion image (MOTI) can represent an image in which the movement of an image changes due to the movement of a subject.

[0021] The disparity estimation unit (120) can determine the size of the variation in each position of the image based on the long exposure image (L), short exposure image (S), extreme exposure images (VS1, VS2) and noise profile value (NP) applied from the image capture unit (105). The disparity estimation unit (120) can generate a disparity value (DS) based on the size of the variation in each position of the image and the noise profile value (NP).

[0022] That is, the disparity estimation unit (120) can detect the variation (difference in exposure ratio) of the image by position based on the long exposure image (L), the short exposure image (S), and the extreme exposure images (VS1, VS2). That is, the disparity estimation unit (120) can store information about the variation (difference in exposure ratio) at a specific position of the image generated by the image sensor when the vehicle is driving. For example, the disparity estimation unit (120) can measure the degree to which the exposure ratio (sensitivity ratio) set to the sensor is different in an image having multiple exposures (or sensitivities) to calculate the amount of variation in the image at a specific position. In addition, the disparity estimation unit (120) can compare the variation size of the image by position and the noise profile value (NP) and output the disparity value (DS) corresponding to the difference value.

[0023] For example, if the size of the image fluctuation (difference in exposure ratio) at a specific location is smaller than the noise profile value (NP), the disparity estimation unit (120) can determine that the fluctuation is due to noise and ignore the fluctuation. That is, the disparity estimation unit (120) can output the disparity value (DS) as 0.

[0024] The weight application unit (125) can calculate the motion weight (WM) used for synthesizing the LFM image (LFMI) and the motion image (MOTI). The weight application unit (125) can set the position-specific fluctuation probability of the image, i.e., the motion weight (WM), by using the vehicle's driving speed (SP), the curvature of the lens and the distance from the center of the image (D), the exposure time (T), and the motion probability information (P) for each region of the image. The operation of the weight application unit (125) will be described in more detail in FIG. 4 described below.

[0025] The blending unit (130) can generate a composite image (BI) by synthesizing an LFM image (LFMI) and a motion image (MOTI) based on a disparity value (DS) and a motion weight (WM). The blending unit (130) can synthesize the LFM image (LFMI) and the motion image (MOTI) using a two-step weighted sum algorithm. For example, the blending unit (130) can synthesize the LFM image (LFMI) and the motion image (MOTI) by applying a motion weight (WM), and can generate the composite image (BI) by applying a disparity value (DS) to the image synthesized with the motion image (MOTI). The operation of the blending unit (130) will be described in more detail later with reference to FIG. 5.

[0026] The tone mapping application unit (135) can apply tone mapping to a composite image (BI) to generate an HDR image (HDRI). “Tone mapping” may be referred to as adjusting brightness and color to output an image with a wide dynamic range on an output interface with a low dynamic range (e.g., an LDR (low dynamic range) display). Applying tone mapping may be referred to as applying a tone mapping algorithm (e.g., a tone mapping algorithm using sigmoid and / or histogram).

[0027] Motion images (MOTI) may have motion blur, but they can be HDR images with sufficient dynamic range (DR). LFM images (LFMI) may lack dynamic range (DR) for automotive use. Therefore, it is desirable to output LFM images (LFMI) in areas where LED flicker occurs, and motion images (MOTI) in all other areas. If motion images (MOTI) are used in areas where LED flicker occurs, the LED flicker may not be removed. On the other hand, if LFM images (LFMI) are used in areas where motion-induced image changes occur without LED flicker, the dynamic range may not be sufficiently expressed.

[0028] In static areas (no movement between exposures) where LED flicker or motion is not present, the brightness ratios of the captured long-exposure, short-exposure, and extreme-exposure images may be equal to the exposure ratio set on the sensor. Therefore, areas where the brightness ratio does not equal the exposure ratio can be determined to be experiencing motion or LED flicker. However, it can be very difficult to determine whether the difference in brightness ratio from the exposure ratio is due to motion or LED flicker.

[0029] Accordingly, an embodiment of the present invention can express a wider dynamic range by more accurately adjusting the synthesis ratio of an image for LED flicker mitigation and a motion image when a change (e.g., a difference in exposure ratio) occurs at a specific location in a plurality of images captured by an imaging device (100).

[0030] Figure 2 is a detailed configuration diagram of the image capture unit of Figure 1.

[0031] Referring to FIG. 2, the image capture unit (105) may include a pixel array (210), an exposure control circuit (220), a gain control unit (230), an analog-to-digital signal converter (hereinafter, referred to as 'ADC') (240), a noise profiling unit (250), and / or an interface (260).

[0032] For example, the image capture unit (105) can collect an optical signal through a pixel array (210) and convert it into an electrical signal. The pixel array (210) can include a plurality of pixels (described later). Each pixel can convert an optical signal into an electrical signal according to the photoelectric effect. Each pixel can include a photodiode. The pixel array (210) can be configured in a matrix form in which a plurality of pixels are arranged in rows and columns. Each pixel can be accessed and / or referenced through a unique row and column address.

[0033] The image capture unit (105) can adjust the exposure time by controlling the time at which each pixel collects charge through the exposure control circuit (220) when converting an optical signal into an electrical signal through the pixel array (210).

[0034] The image capture unit (105) can apply an analog gain through the gain control unit (230) before converting the converted electrical signal into a digital signal. The electrical signal amplified or reduced by applying the analog gain can be converted into a digital signal through the analog-to-digital signal converter (240) of the image capture unit (105).

[0035] In addition, the noise profiling unit (250) can profile the noise of the image capture unit (105) to generate a noise profile value (NP) and transmit it to the disparity estimation unit (120). The size of the change in the position of the image can be caused by noise in addition to LED flicker or motion. Accordingly, the noise profiling unit (250) can express the noise level of the image sensor as a linear function of brightness. The noise profiling unit (250) can model the noise profile value (NP) as in [Mathematical Formula 1] below.

[0036] [Mathematical Formula 1]

[0037]

[0038] In the above [Mathematical Formula 1], x represents the signal level of the image, A models noise that is dependent on the image signal, and B may correspond to a model of noise that is independent of the image signal. Here, the values ​​of A and B are determined by the noise characteristics of the sensor, and the values ​​can be determined by measuring the noise characteristics of the actual sensor or by tuning the sensor.

[0039] The digital signal converted by the ADC (240) can be transmitted to the first synthesis unit (110), the second synthesis unit (110), and the disparity estimation unit (120) through the interface (260) of the image capture unit (105). In addition, the noise profile value (NP) generated by the noise profiling unit (250) can be transmitted to the disparity estimation unit (120) through the interface (260).

[0040] Fig. 3 is a configuration diagram of the pixel array of Fig. 2.

[0041] Image sensors used in the automotive industry may require high accuracy and high performance to ensure driver safety and support autonomous driving functions. In particular, the latest autonomous driving technologies and safety systems may require accurate and rapid information about the surrounding environment. Consequently, advanced image sensor technologies, such as LED Flicker Mitigation (LFM) and High Dynamic Range (HDR), may be required.

[0042] Here, LFM technology can be primarily used to mitigate flicker caused by LED light sources. Especially with the increasing use of LED headlights in vehicles, rapidly changing brightness fluctuations from light sources can affect image sensors. These brightness fluctuations can compromise the accuracy of recognizing traffic signals, such as traffic lights and road signs. Therefore, LFM technology can help image sensors accurately recognize information by mitigating rapidly changing brightness fluctuations from light sources.

[0043] Furthermore, HDR technology can play a crucial role in ensuring that vehicle cameras capture accurate images in road conditions with varying lighting conditions. HDR technology is necessary for image sensors to simultaneously capture details in both bright and dark areas, both in dark environments like tunnels and dark roads, and in bright sunlight. This allows autonomous driving systems to accurately perceive their surroundings in diverse environments.

[0044] However, supporting both LFM and HDR technologies on a single image sensor may not be easy, as they require opposite exposure characteristics during image capture. LFM requires a long exposure time to detect and mitigate rapidly flickering lights. This is because the sensor must be exposed for a longer time to detect the brightness of the flickering lights. Furthermore, HDR requires appropriately adjusted exposure times to accurately capture both bright and dark areas under various lighting conditions. In other words, short exposure times are required to express bright areas, and long exposure times are required to express dark areas.

[0045] Therefore, simultaneous implementation of LFM and HDR technologies presents challenges due to conflicting factors in terms of exposure time control. To address these conflicting factors in a single sensor, low-sensitivity, small pixels can be used for LFM technology.

[0046] FIG. 3 may show an example of a pixel array (210) structure of an HDR imaging device (100) supporting LFM technology.

[0047] Referring to FIG. 3, the pixel array (210) may include a plurality of pixels (LPX1 to LPX4, SPX1 to SPX4). For example, the plurality of pixels (LPX1 to LPX4, SPX1 to SPX4) may be two-dimensionally arranged in a matrix form. The plurality of pixels (LPX1 to LPX4, SPX1 to SPX4) may be regularly arranged in a first direction (X) and a second direction (Y).

[0048] The pixel array (210) may be implemented as a unit pixel (UP) including two types of photodiodes with different sensitivities to satisfy a high dynamic range. Here, the unit pixel (UP) may be a unit of pixels that receive light and output an image corresponding to one pixel. Among the plurality of pixels (LPX1 to LPX4, SPX1 to SPX4), the pixels (LPX1 to LPX4) may correspond to large pixels, and the pixels (SPX1 to SPX4) may correspond to small pixels.

[0049] For example, the pixel array (210) may include one large pixel (LPX1) and one small pixel (SPX1) to constitute one unit pixel (UP). The sensitivities of the large pixel (LPX1) and the small pixel (SPX1) may be different. For example, the sensitivity of the small pixel (SPX1) may be made small, and the large pixel (LPX1) may be implemented to have a greater sensitivity than the small pixel (SPX1). The area of ​​the large pixel (LPX1) may be larger than the area of ​​the small pixel (SPX1). That is, the amount of light incident on the large pixel (LPX1) may be greater than the amount of light incident on the small pixel (SPX1). The unit pixels (UP) corresponding to the large pixel (LPX1) and the small pixel (SPX1) may convert light to generate an electrical signal.

[0050] The large pixel (LPX1) and the small pixel (SPX1) can be distinguished in shape when viewed from the top. The large pixel (LPX1) may have an octagonal shape as illustrated, and the small pixel (SPX1) may have a square shape as illustrated, but the embodiment of the present invention is not limited thereto. The large pixel (LPX1) and the small pixel (SPX1) may contact each other. Similarly, the remaining large pixels (LPX2 to LPX4) and small pixels (SPX2 to SPX4) may also be paired to form a unit pixel.

[0051] By arranging the large pixels (LPX1 to LPX4) and the small pixels (SPX1 to SPX4) so ​​that they do not overlap each other, light receiving efficiency can be optimized, and by arranging the small pixels (SPX1 to SPX4) in the remaining space after arranging the large pixels (LPX1 to LPX4), the integration degree can be improved. In addition, although FIG. 3 illustrates arranging the small pixels (SPX1 to SPX4) to the lower right of the large pixels (LPX1 to LPX4), the scope of the present invention is not limited thereto, and arranging them at other adjacent positions is also possible.

[0052] In bright daylight conditions, image sensor exposures are typically very short, making LED flicker a common problem. Mitigating LED flicker typically requires exposure times longer than a certain amount of time. However, longer exposures can saturate the image, resulting in information loss. To prevent this problem, small, low-sensitivity pixels can be used for LFM applications.

[0053] To generate an HDR image, multiple exposure images are used, and by combining multiple photos captured at different exposures, an image with a wider dynamic range (DR) can be expressed. An HDR imaging device (100) can capture and synthesize three exposure images, namely a long exposure image (L), a short exposure image (S), and extreme exposure images (VS1, VS2), to generate an HDR image (HDRI).

[0054] The image capture unit (105) can generate a long exposure image (L), a short exposure image (S), and an extreme exposure image (VS1) using small pixels (e.g., SPX1) from a pixel array (210) having a structure as shown in FIG. 3. In addition, the image capture unit (105) can generate a long exposure image (L), a short exposure image (S), and an extreme exposure image (VS2) using large pixels (e.g., LPX1) from a pixel array (210) having a structure as shown in FIG. 3.

[0055] Fig. 4 is a drawing for explaining the operation of the weight application unit of Fig. 1.

[0056] LFM technology can be a crucial feature in road environments. HDR image sensors using LFM technology can misidentify fluctuations (differences in exposure ratio) that occur while a vehicle is moving as LED flicker. However, images of stationary vehicles can be considered static because motion cannot occur. Therefore, fluctuations in this situation can be attributed to LED flicker, not motion.

[0057] Of course, even when the car is stationary, certain objects within the image may move, but it's safe to assume that all of these motions are not caused by LED flicker. In other words, the most important objects requiring LFM consideration are those without motion, such as traffic lights or signs.

[0058] When a vehicle is moving, the magnitude of motion observed in a specific area of ​​the screen can vary depending on the position of the image due to the lens effect. Even when the vehicle is moving very fast, there may be little motion in the center of the image. Wide-angle lenses are often used for vehicles, and wide-angle lenses have little motion in the center of the image, while motion-induced fluctuations can increase toward the edges of the image. Therefore, it is safe to assume that fluctuations in the center of the image during a vehicle's movement are primarily due to LED flicker, while fluctuations at the edges of the image are primarily due to motion.

[0059] Additionally, if the exposure is long enough that LED flicker does not occur, any fluctuations occurring in the image can be judged as motion. That is, the fluctuations in the position of the image (differences in exposure ratio) occurring in the image sensor when the vehicle is driving can be interpreted as fluctuations due to LED flicker or motion depending on the vehicle's driving speed (S), the curvature of the lens and the distance from the center of the image (D), the motion probability information (P) for each area of ​​the image (e.g., 8×8 block), and the exposure time (T).

[0060] Accordingly, the weight application unit (125) can set a motion weight (WM) by calculating the vehicle's driving speed (SP), the curvature of the lens and the distance from the center of the image (D), the exposure time (T), and the motion probability information (P) for each region of the image.

[0061] For example, the exposure time (T) can be calculated during the exposure process of the image sensor. The vehicle's driving speed (SP) can be calculated based on values ​​monitored during the vehicle's driving process. The distance (D) from the center of the lens to the corresponding image location can be calculated based on the characteristics of the lens attached to the image sensor. The motion probability information (P) for each region of the image can be set in a lookup table (LUP).

[0062] The weight application unit (125) can calculate the motion weight (WM) using a function that takes the driving speed (SP) of the vehicle as a factor, a function that takes the position (D) according to the angle of view of the lens as a factor, a function that takes the exposure time (T) of the camera as a factor, and a function that takes the motion probability information (P) for each region of the image as a factor, as shown in [Mathematical Formula 2] below.

[0063] [Equation 2]

[0064]

[0065] In the above [Mathematical Formula 2], the motion weight (WM) can be calculated by multiplying the weight (Ws) by the vehicle's driving speed (SP), the weight (Wd) by the distance (D) from the center of the lens to the corresponding image position, the weight (Wt) by the exposure time (T), and the weight (Wp) by the motion probability (P) for each region of the image.

[0066] In the present disclosure, the weight application unit (125) can calculate the motion weight (WM) by using all or part of the weights (Ws, Wd, Wt, Wp), and the above [Mathematical Formula 2] can be sufficiently changed depending on the application of the weights.

[0067] For example, instead of using weights (Wt, Wd) according to exposure time (T) and distance (D), only the threshold value (Ws) according to the vehicle's driving speed (SP) can be used in the motion weighting (WM) calculation to increase the probability of detection due to LED flicker when the vehicle is stopped. In some embodiments, the characteristics for the vehicle's driving speed (SP) and the distance (D) according to the lens's field of view can be obtained by predicting motion for each image area (position) using an algorithm such as dense optical flow.

[0068] The function for calculating each weight (Ws, Wd, Wt, Wp) in the threshold application unit (125) has the characteristics as illustrated in FIG. 4. Referring to FIG. 4, the smaller each weight value (Ws, Wd, Wt, Wp) is, the higher the ratio of the LFM image (LFMI) can be. For example, when the range of each weight value (Ws, Wd, Wt, Wp) is defined as 0 to 1, the value of each weight value (Ws, Wd, Wt, Wp) can mean the probability that the cause of a change in a specific location of the image is not due to LED flicker.

[0069] The longer the exposure time (T) at which a change occurs at a specific location in the image, the more likely it is that the change is not due to LFM imaging (LFMI). In particular, if the exposure time (T) is longer than a preset threshold time (TH), the weight (Wt) can be maximized because LFM is unlikely to occur.

[0070] Looking at the weights (Ws), when a change occurs at a specific location in the image, the higher the driving speed (SP), the more likely it is that the change is not due to an LFM image (LFMI). Looking at the weights (Wd), when a change occurs at a specific location in the image, the further the location in the image from the center of the lens, the more likely it is a motion image (MOTI) rather than an LFM image (LFMI). This depends on the field of view characteristics of the lens, and the longer the distance, the greater the likelihood of a motion over time image (MOTI). For example, if the lens is a wide-angle lens, the weight (Wd) can be modeled as a function that increases more rapidly with distance (D).

[0071] The motion probability (P) for each region of the image can be used to output motion images (MOTI) by setting areas where LED light sources cannot exist, depending on the camera installation environment. The weight (Wp) based on the motion probability (P) for each region of the image can also be used for purposes similar to the distance (D) described above.

[0072] The form of these weighting functions and the formula for motion weights (WM) using these weighting functions are briefly explained with examples to facilitate understanding of the present invention. Therefore, the characteristic functions illustrated in Fig. 4 are merely examples to aid understanding, and it is desirable to optimize the actual, precise function through experiments along with the variation in the disparity value (DS).

[0073] Figure 5 is a detailed configuration diagram of the blending section of Figure 1.

[0074] Referring to FIG. 5, the blending unit (130) may include a first image synthesis unit (510) and a second image synthesis unit (520).

[0075] The first image synthesis unit (510) can synthesize the LFM image (LFMI) and the motion image (MOTI) by applying a motion weight (WM) and generate a synthetic image (ML). The first image synthesis unit (510) can generate the synthetic image (ML) by performing a weighted sum on the LFM image (LFMI) and the motion image (MOTI) as in the following [Mathematical Formula 3].

[0076] [Equation 3]

[0077]

[0078] As in [Mathematical Formula 3] above, a synthetic image (ML) can be generated by adding the value obtained by multiplying the motion image (MOTI) by the motion weight (WM) and the value obtained by multiplying (1-WM) by the LFM image (LFMI).

[0079] For example, the value of the motion weight (WM) can range from 0 to 1. The first image synthesis unit (510) can generate a synthetic image (ML) by setting the LFM image (LFMI) to 100% when the value of the motion weight (WM) is 0. In addition, the blending unit (130) can generate a synthetic image (ML) by setting the motion image (MOTI) to 100% when the value of the motion weight (WM) is 1.

[0080] HDR image sensors that support LFM technology must determine whether changes in position in an image are caused by LED flicker or motion. The value of the motion weighting (WM) can indicate the probability that the difference in exposure ratio in an image is not caused by LED flicker. For example, the larger the value of the motion weighting (WM), the more likely it is that the difference in exposure ratio in an image is caused by motion, and the smaller the value of the motion weighting (WM), the more likely it is that the difference in exposure ratio in an image is caused by LED flicker.

[0081] And, the second image synthesis unit (520) can synthesize the motion image (MOTI) and the synthesis image (ML) by applying the disparity value (DS) and generate the synthesis image (BI). The second image synthesis unit (520) can generate the synthesis image (BI) by performing a weighted sum on the motion image (MOTI) and the synthesis image (ML) as in the following [Mathematical Formula 4].

[0082] [Equation 4]

[0083]

[0084] As in [Mathematical Formula 4] above, the synthetic image (BI) can be generated by adding the value obtained by multiplying the synthetic image (ML) by the disparity value (DS) and the value obtained by multiplying (1-DS) by the motion image (MOTI). That is, the second image synthesis unit (520) can synthesize the ML by reflecting it more in the synthetic image (ML) as the disparity value (DS) increases, and can synthesize the ML by reflecting it more in the motion image (MOTI) as the disparity value (DS) decreases.

[0085] The above description is merely an example of the technical idea of ​​the present invention, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present invention.

[0086] Accordingly, the embodiments disclosed in the present invention are intended to illustrate, rather than limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical concepts within the scope equivalent thereto should be construed as being included within the scope of the present invention.

Claims

1. An image capture unit that captures multiple images with different exposure values ​​to generate a long exposure image, a short exposure image, a first extreme exposure image, and a second extreme exposure image; A first synthesizing unit that generates an image for alleviating LED flicker by synthesizing the long exposure image, the short exposure image, and the first extreme exposure image; A second synthesizing unit that generates a motion image by synthesizing the long exposure image, the short exposure image, and the second extreme exposure image; and An imaging device that generates a wide dynamic range image, including a blending unit that generates a composite image by synthesizing the LED flicker mitigation image and the motion image based on a disparity value indicating a difference in exposure ratio by position of the image and a motion weight.

2. In claim 1, the image capture unit A pixel array comprising a plurality of unit pixels, which collects optical signals and converts them into electrical signals; and An imaging device that generates a wide dynamic range image, comprising a noise profiling unit that generates a noise profile value by profiling noise in a plurality of images having different exposure values.

3. In claim 2, Each of the plurality of unit pixels includes a large pixel and a small pixel having a lower sensitivity than the large pixel, An imaging device that generates a wide dynamic range image, wherein the large pixel generates a wide dynamic range image having a larger area than the small pixel.

4. In claim 3, the image capture unit Generating the first extreme exposure image using the small pixels, An imaging device that generates a wide dynamic range image by using the large pixels to generate the second extreme exposure image.

5. In claim 2, Detecting the exposure ratio for each position of the image based on the long exposure image, the short exposure image, the first extreme exposure image, and the second extreme exposure image, An imaging device for generating a wide dynamic range image, further comprising a disparity estimation unit that compares the exposure ratio and the noise profile value and generates the disparity value corresponding to the difference value.

6. In claim 5, the disparity estimation unit An imaging device that generates a wide dynamic range image that outputs the disparity value as '0' when the size of the exposure ratio by position of the image is smaller than the noise profile value.

7. In claim 1, An imaging device for generating a wide dynamic range image further comprising a weighting unit that sets the motion weight using at least one of the driving speed of the vehicle, the curvature of the lens, the distance from the center of the image, the exposure time, and the motion probability information for each region of the image.

8. In claim 1, the blending unit A first image synthesis unit that synthesizes the LED flicker mitigation image and the motion image based on the motion weight to output a first image; and An imaging device for generating a wide dynamic range image, comprising a second image synthesis unit for generating the composite image by synthesizing the first image and the motion image based on the disparity value.

9. In claim 8, the first image synthesis unit If the value of the above motion weight is the first value, the ratio of the image for LED flicker mitigation is increased to synthesize the image. An imaging device that generates a wide dynamic range image by synthesizing an image by increasing the ratio of the motion image when the value of the motion weighting is a second value greater than the first value.

10. In claim 8, the second image synthesis unit If the above disparity value is the first value, the ratio of the motion image is increased to synthesize the image. An imaging device that generates a wide dynamic range image by synthesizing an image by increasing the ratio of the first image when the disparity value is a second value greater than the first value.

11. In claim 1, An imaging device for generating a wide dynamic range image further comprising a tone mapping application unit for generating an HDR image by applying tone mapping to the above-mentioned synthetic image.

Images