Automotive imaging system including an electronic image sensor with a sparsely populated color filter array

Abstract

Motor vehicle, which includes the following: a frame and an imaging system (10) supported by the frame and comprising the following: a camera (100) including a lens (110) and an image sensor (120) with a sparsely populated color filter array, wherein the camera (100) is configured to to generate image data based on light incident on the image sensor (120), and the lens (110) has an aperture higher than f / 2.4; a controller (20) configured to generate an image using the image data; and a display device (30) configured to display the image, characterized by the fact that the image sensor (120) contains several pixels and the sparsely populated color filter array contains a repeating mosaic (130) of color filters arranged in a pattern such that the mosaics (130) are more concentrated near a center point of the image sensor (120) than near the edges of the image sensor (120).

Description

TECHNICAL AREA

[0001] The present disclosure relates generally to a motor vehicle imaging system. In particular, the motor vehicle imaging system includes a camera with a lens and an electronic image sensor with a sparse color filter array. BACKGROUND

[0002] Cameras typically used in known automotive imaging systems include an electronic image sensor, such as an active pixel sensor (APS) manufactured using a CMOS (complementary metal-oxide semiconductor) process, or a CCD (charge-coupled device). This known electronic image sensor contains multiple light-sensitive pixels, typically arranged in a matrix of rows and columns. Fig. Figure 1A represents an exemplary electronic image sensor 12 according to the prior art, comprising 144 pixels 12a arranged in a 12×12 matrix. In operation, the circuit of this known electronic image sensor converts light incident on a pixel into digital image data. A controller converts the image data into an image that can be displayed on a display device.

[0003] Document US 2007 / 0159542A1 describes an imaging device comprising a sparsely populated color filter array, wherein a camera is configured to generate image data based on light falling on a sensor, and a controller is configured to generate an image using the image data.

[0004] To capture color data and enable the creation of color images, these well-known electronic image sensors incorporate a Bayer color filter array (CFA). The Bayer CFA includes red, green, and blue color filters arranged or formed in a repeating pattern over the pixels (one per pixel). Fig. Figure 1B represents the prior art electronic image sensor 12, which incorporates a Bayer CFA including a repeating mosaic 13 consisting of two green filters 13a, one red filter 13b, and one blue filter 13c. In operation, each color filter allows only light of a specific wavelength (red, green, or blue in this case) to reach its pixel, thus providing color information to the electronic image sensor. The controller uses a demosaicing algorithm to reconstruct the full-color image using this color information.

[0005] Since each color filter allows only light of a specific wavelength to reach its pixel, approximately two-thirds less light reaches each filtered pixel than if the pixel were unfiltered. Using a Bayer CFA—where each pixel is filtered—results in a significantly reduced light sensitivity. In turn, this leads to a lower-quality color image compared to a monochrome image obtained using an unfiltered electronic image sensor. This problem is exacerbated when used in low-light conditions. For example, the color image may exhibit increased noise, aliasing, reduced spatial resolution, and decreased sharpness, caused by the reduced light sensitivity or by post-processing techniques used to compensate for it.

[0006] There is a need for new automotive imaging systems that solve these problems. SUMMARY

[0007] The attached claims define this application. The description summarizes aspects of the embodiments and should not be used to limit the claims. Other implementations are considered in accordance with the techniques described herein, as will be apparent to a person skilled in the art upon examination of the following drawings and the detailed description, and these implementations are intended to be within the scope of protection of this application.

[0008] Exemplary embodiments provide a motor vehicle and a motor vehicle imaging system for the motor vehicle. According to one embodiment, a motor vehicle comprises a frame and an imaging system supported by the frame, comprising: a camera including a lens and an image sensor with a sparse color filter array, wherein the camera is configured to generate image data based on the light incident on the sensor, and the lens has an aperture greater than f / 2.4; a controller configured to generate an image using the image data; and a display device configured to display the image.

[0009] According to another embodiment, a motor vehicle imaging system comprises a camera including a lens and an image sensor with a sparse color filter array, wherein the camera is configured to generate image data based on the light incident on the sensor and the lens has an aperture greater than f / 2.4; a controller configured to generate an image using the image data; and a display device configured to display the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For a better understanding of the invention, the detailed description refers to embodiments illustrated in the following drawings. The components in the drawings are not necessarily to scale, and related elements may be omitted to emphasize and clearly illustrate the novel features described herein. Additionally, system components can be arranged in various ways, as is known from the art. In the drawings, identical reference numbers in different figures may consistently refer to the same parts unless otherwise specified. Fig. 1A represents a state-of-the-art electronic image sensor. Fig. 1B represents the electronic image sensor of Fig. 1A with a Bayer CFA. Fig. Figure 2 is a block diagram of an embodiment of the vehicle imaging system of the present disclosure. The Fig. 3A, Fig. 3B and Fig. 3C represent different embodiments of electronic image sensors with different types of sparsely populated CFAs. DETAILED DESCRIPTION OF EXAMPLES Automotive imaging system including an electronic image sensor with a sparsely populated color filter array

[0011] Although the vehicle imaging system of the present disclosure may be embodied in various forms, the drawings represent some exemplary and non-limiting embodiments of the vehicle imaging system, and this description describes these. The present disclosure is an illustration of the vehicle imaging system and does not limit the vehicle imaging system to the specific embodiments illustrated and described. It is possible that not all of the illustrated and described components are necessary, and some embodiments may include additional, different, or fewer components. The arrangement and nature of the components may vary without departing from the concept or scope of the claims set forth herein.

[0012] Various embodiments of the automotive imaging system of the present disclosure (sometimes referred to for brevity as the “imaging system”) include an electronic image sensor with a sparsely populated CFA. Depending on the implementation, the use of a sparsely populated CFA as opposed to a Bayer CFA results in: (1) improved image quality and / or (2) cost savings by enabling the use of a less complex, less expensive lens without a reduction or significant reduction in image quality. 1. Automotive imaging system components

[0013] Fig. Figure 2 is a block diagram of an embodiment of a vehicle imaging system 10 of the present disclosure. In this embodiment, the vehicle imaging system 10 is supported by the frame of a motor vehicle, although in other embodiments the vehicle can be any suitable vehicle (such as a truck, a watercraft, or an aircraft). The vehicle imaging system 10 includes a camera 100, a controller 20, and a display unit 30. The camera 100 and the display unit 30 are electrically connected to the controller 20 so that the controller 20 can send signals to and receive signals from these components (and vice versa).

[0014] The camera 100 can be any suitable camera that includes a lens 110 and an electronic image sensor 120. The lens 110 can be any suitable lens, such as (among others) a rectilinear lens or an automotive fisheye lens (also called a wide-field lens). The lens can be a fixed-aperture or variable-aperture lens. The electronic image sensor 120 can be any suitable electronic image sensor—such as an APS or a CCD—that includes multiple pixels 120a arranged in a matrix of rows and columns. The electronic image sensor 120 is positioned relative to the lens 110 such that the lens focuses incident light onto the pixels 120a.

[0015] The electronic image sensor incorporates a thin-cell CFA (SCFA), which contains fewer color filters than a Bayer CFA with the same pixel density. With all other factors kept constant, the use of the SCFA increases the amount of light reaching the pixels compared to a Bayer CFA.

[0016] As in Fig. As shown in Figure 3A, the electronic image sensor 120 in this embodiment includes a 3×3 SCFA with a repeating mosaic 130 consisting of two green filters 130a, one red filter 130b, and one blue filter 130c. In the 3×3 SCFA, the mosaics 130 are arranged in a pattern such that a row or column of unfiltered pixels 120 separates adjacent mosaics 130 from each other.

[0017] Fig. 3B represents another embodiment of an electronic image sensor 1120, which includes a 4×4 SCFA with a repeating mosaic 1130 consisting of two green filters 1130a, one red filter 1130b, and one blue filter 1130c. In the 4×4 SCFA, the mosaics 1130 are arranged in a pattern such that two rows or columns of unfiltered pixels 1120 separate adjacent mosaics 1130 from one another.

[0018] Fig.3C presents another embodiment of an electronic image sensor 2120, which includes a fisheye SCFA with a repeating mosaic 2130 consisting of two green filters 2130a, one red filter 2130b, and one blue filter 2130c. In the fisheye SCFA, the mosaics 2130 are arranged in a pattern such that the mosaics 2130 are close together near the horizontal and vertical center of the electronic image sensor 2120 and further apart away from the center. This results in a high concentration of mosaics near the center of the electronic image sensor and a low concentration of mosaics near the edges of the electronic image sensor. This spacing corresponds to the unique field of view of a fisheye lens.

[0019] These three SCFAs are merely examples, and the electronic image sensor can incorporate any suitable SCFA with any suitable mosaics arranged in any suitable patterns, with any suitable color filters, and with any suitable amount of filters, as long as certain pixels remain unfiltered.

[0020] The controller 20 includes at least one processor that communicates with a memory that stores a set of instructions. The processor is configured to communicate with the memory, access the set of instructions, and execute the set of instructions, causing the imaging system 10 to perform any of the procedures, processes, and features described herein.

[0021] The processor can be any suitable processing unit or set of processing units, such as, but not limited to, a microprocessor, a microcontroller-based platform, a suitable integrated circuit, or one or more application-specific integrated circuits (ASICs) configured to execute the set of instructions. The memory can be any suitable memory, such as, but not limited to, random-access memory (RAM), which may include non-volatile RAM, magnetic RAM, ferroelectric RAM, and any other suitable forms; disk storage; a hard disk drive (HDD); a solid-state drive (SSD); flash memory, EPROMs; EEPROMs; memristor-based non-volatile semiconductor memory; immutable memory; or read-only memory.

[0022] The display device 30 can be any suitable display device configured to display images generated by the controller 20, such as (among others) a liquid crystal display (LCD) or an organic light-emitting diode (OLED). The display device 30 can be of any suitable size and shape and can be positioned at any suitable location within the vehicle. In one embodiment, the display device is part of the vehicle's infotainment system. 2. Operation

[0023] To generate and display an image using the automotive imaging system 10, the camera 100 first exposes the electronic image sensor 120 (and in particular the pixels 120a) (via the lens 110) to incident light for a period of time called the integration time. The camera 100 accomplishes this by opening its shutter, leaving the shutter open for the integration time, and then closing the shutter. The shutter can be electronic or mechanical. Circuitry within the electronic image sensor 120 converts the incident light onto the pixels into digital image data and sends the image data to the controller 20. The controller 20 converts the image data into an image—at least partially using a demosaicing algorithm known in the art for restoring color—and sends the image to the display unit 30 for display. 3. Aperture and integration time

[0024] As is well known in engineering, a camera lens has an f-number (sometimes called an aperture number or f-ratio), which represents the ratio of the lens's focal length to the diameter of its entrance pupil. The smaller the f-number, the larger the aperture and the more light passes through the lens, and vice versa. For example, the aperture of an f / 1.4 lens is larger than that of an f / 8 lens. Lenses with larger f-numbers (smaller apertures) have a greater depth of field than those with smaller f-numbers (larger apertures), which is advantageous for applications where it is desired that the resulting images convey depth. Lenses with smaller f-numbers (larger apertures) are more complex than lenses with larger f-numbers (smaller apertures), meaning they are housed in larger bodies.

[0025] Producing an image of acceptable quality requires balancing three main factors: (1) the aperture of the lens, (2) the integration time, and (3) the type of CFA (if any) of the electronic image sensor.

[0026] Table 1 below shows, for each of several f-numbers and for four types of CFAs (none, 4×4-SCFA, 3×3-SCFA and Bayer-CFA), the percentage of incident light reaching the pixels of the electronic image sensor, normalized to f / 2.4 and assuming an integration time of 30 milliseconds.

[0027] Table 2 below shows, for each of several integration times and for four types of CFAs (none, 4×4-SCFA, 3×3-SCFA and Bayer-CFA), the percentage of incident light reaching the pixels of the electronic image sensor, normalized to any f-number.

[0028] Typical automotive imaging systems include an electronic image sensor with a Bayer CFA, a lens with an f / 2.4 aperture, and an integration time of 30 milliseconds. This results in 33.33% of the incident light reaching the pixels of the electronic image sensor.

[0029] As shown in Tables 1 and 2, the quality of the image ultimately produced by the controller can be improved by using an electronic image sensor with the 3×3 SCFA or the 4×4 SCFA of this disclosure and keeping the integration time and aperture constant (30 milliseconds and f / 2.4 in this embodiment) compared to that produced when using the Bayer CFA. Using the 3×3 SCFA increases the percentage of incident light reaching the pixels of the electronic image sensor by more than 2 times to 69%, and using the 4×4 SCFA increases the percentage of incident light reaching the pixels of the electronic image sensor by more than 2.5 times to 86%. Since the integration time and aperture remain unchanged, more light reaching the pixels results in a higher quality image. This also improves low-light performance.Because more light reaches the pixels when an electronic image sensor with the 3×3-SCFA or the 4×4-SCFA is used, the automotive imaging system can operate acceptably in low-light conditions and in some cases does not require the vehicle to generate its own light source to provide an acceptable amount of ambient light.

[0030] As shown in Table 1, by using an electronic image sensor with the 3×3 SCFA or the 4×4 SCFA of the present disclosure and keeping the integration time constant (30 milliseconds in this embodiment), a higher f-number can be used compared to the f-number used with the Bayer CFA without degradation or significant degradation in image quality. For example, using an electronic image sensor with the 3×3 SCFA combined with an f / 2.8 lens—one f-number higher than the f / 2.4 lens used with the Bayer CFA—results in 48.3% of the incident light reaching the pixels of the electronic image sensor. In this case, using a less complex lens with a higher f-number results in a more compact camera and allows more light to reach the pixels than in the typical automotive imaging systems described above.And since the integration time remains constant, more light reaching the pixels results in a higher quality image. Furthermore, a higher f-number provides greater depth of field and therefore a sharper image across the image plane.

[0031] Using an electronic image sensor with the 4×4 SCFA combined with an f / 4 lens—two f-stops faster than the f / 2.4 lens used with the Bayer CFA—results, in another example, in 30.1% of the incident light reaching the pixels of the electronic image sensor. In this case, using a less complex lens with a higher f-number results in a more compact camera and allows approximately the same amount of light to reach the pixels as in the typical automotive imaging systems described above. Accordingly, the image quality will not be substantially worse, but a less complex, cheaper, and smaller lens can be used. Furthermore, a higher f-number provides greater depth of field and therefore a sharper image across the image plane.

[0032] As shown in Table 2, by using an electronic image sensor with the 3×3 SCFA or the 4×4 SCFA of this disclosure and keeping the f-number constant (f / 2.4 in this embodiment), a shorter integration time can be used compared to the integration time used with the Bayer CFA, and the image quality can be improved. For example, using an electronic image sensor with the 3×3 SCFA and an integration time of 15 milliseconds—half that used with the Bayer CFA—results in 34.5% of the incident light reaching the pixels of the electronic image sensor. In this case, the reduction in integration time leads to a sharper, higher-quality image. And since the f-number does not change, more light reaching the pixels also results in a higher-quality image.

[0033] Using an electronic image sensor with the 4×4 SCFA and an integration time of 15 milliseconds—half that used with the Bayer CFA—results, in another example, in 43% of the incident light reaching the pixels of the electronic image sensor. In this case, the reduction in integration time leads to a sharper, higher-quality image. And since the aperture remains unchanged, more light reaching the pixels also results in a higher-quality image.

[0034] It is therefore understood that an aperture number, an integration time and a type of SFCA can be selected to produce a sharper image, improved light uniformity across the entire image (reduced vignetting), improved color reproduction, improved signal-to-noise ratio and improved dynamic range and improved hue reproduction.

[0035] The embodiments described above, and in particular all "preferred" embodiments, are possible examples of implementations and are presented solely for the purpose of a clear understanding of the principles of the invention. Many variations and modifications can be made to the embodiment(s) described above without substantially deviating from the concept and principles of the techniques described herein. All such modifications are hereby included within the scope of this disclosure and protected by the following claims.

Claims

[1] Motor vehicle comprising: a frame and an imaging system (10) supported by the frame and including: a camera (100) including a lens (110) and an image sensor (120) with a sparse color filter array, the camera (100) being configured to to generate image data based on light incident on the image sensor (120), and the lens (110) has an aperture number higher than f / 2.4; a controller (20) configured to generate an image using the image data; and a display device (30) configured to display the image, characterized by , that the image sensor (120) includes a plurality of pixels and the sparse color filter array includes a repeating mosaic (130) of color filters arranged in a pattern such that the mosaics (130) are more highly concentrated near a center of the image sensor (120) than near the edges of the image sensor (120). [2] The motor vehicle of claim 1, wherein an integration time of the camera (100) is approximately 30 milliseconds or less. [3] Motor vehicle according to claim 1 or claim 2, wherein the aperture of the lens (110) is higher than f / 2.

8. [4] The motor vehicle of claim 3, wherein the sparse color filter array is a 4×4 sparse color filter array and an integration time of the camera (100) is approximately 30 milliseconds. [5] Motor vehicle according to claim 4, wherein the aperture of the lens (110) is f / 4. [6] A motor vehicle according to claim 1, wherein the aperture of the lens (110) is f / 2.8 and the sparse color filter array is a 3×3 sparse color filter array. [7] The motor vehicle of claim 1, wherein an integration time of the camera (100) is approximately 15 milliseconds or less. [8] A motor vehicle according to claim 1, wherein the sparse color filter array is a 3×3 sparse color filter array. [9] Motor vehicle according to one of claims 1 to 8, wherein the lens (110) is a fisheye lens. [10] Automotive imaging system (10) comprising: a camera (100) including a lens (110) and an image sensor (120) with a sparse color filter array, the camera (100) being configured to to generate image data based on light incident on the image sensor (120), and the lens (110) has an aperture number higher than f / 2.4; a controller (20) configured to generate an image using the image data; and a display device (30) configured to display the image, characterized by , that the image sensor (120) includes a plurality of pixels and the sparse color filter array includes a repeating mosaic (130) of color filters arranged in a pattern such that the mosaics (130) are more highly concentrated near a center of the image sensor (120) than near the edges of the image sensor (120). [11] The automotive imaging system (10) of claim 10, wherein an integration time of the camera (10) is approximately 30 milliseconds or less. [12] The automotive imaging system (10) of claim 10 or claim 11, wherein the f-number of the lens (110) is greater than f / 2.

8. [13] The automotive imaging system (10) of claim 12, wherein the sparse color filter array is a 4×4 sparse color filter array and an integration time of the camera (100) is approximately 30 milliseconds. [14] The automotive imaging system (10) of claim 13, wherein the aperture of the lens (110) is f / 4. [15] The automotive imaging system (10) of claim 10, wherein the f-number of the lens (110) is f / 2.8 and the sparse color filter array is a 3×3 sparse color filter array. [16] The automotive imaging system (10) of any one of claims 10 to 15, wherein an integration time of the camera (100) is approximately 15 milliseconds or less. [17] The automotive imaging system (10) of claim 10, wherein the sparse color filter array is a 3×3 sparse color filter array. [18] The automotive imaging system (10) of any one of claims 10 to 17, wherein the lens (110) is a fisheye lens.

Images