System and method for correcting radial color distortion in image captured by image sensor through transflective coating

Abstract

There is provided a rearview mirror assembly for a vehicle, comprising: an electro-optical element having at least one semi-transmissive and semi-reflective coating; at least one imager comprising an array of pixels and positioned behind the electro-optical element to capture an image through the semi-transmissive and semi-reflective coating; a processor capable of: determining correction data for each pixel from a position of the pixel within the array of pixels; and correcting a color value of each pixel of the captured image based on the correction data for the pixel to correct radial color distortion caused by the semi-transmissive and semi-reflective coating.

Description

Technical Field

[0001] The present invention generally relates to an image sensor for imaging the interior of a vehicle, and more particularly to an image sensor that captures images through a transflective coating. Summary of the Invention

[0002] According to one aspect of the present disclosure, a rearview mirror assembly for a vehicle is provided, comprising: an electro-optical element having at least one transflective coating; at least one imager comprising a pixel array and positioned behind the electro-optical element to capture an image through the transflective coating; and a processor capable of: determining correction data for each pixel based on a position of the pixel within the pixel array; and correcting a color value of each pixel of a captured image based on the correction data for the pixel to correct radial color distortion caused by the transflective coating.

[0003] According to another aspect of the present disclosure, a monitoring system for a vehicle is provided, comprising: a transflective coating; at least one imager comprising a pixel array and positioned behind the transflective coating to capture an image through the transflective coating; and a processor capable of: determining correction data for each pixel based on a position of the pixel within the pixel array; and correcting a color value of each pixel of a captured image based on the correction data for the pixel to correct radial color distortion caused by the transflective coating.

[0004] According to another aspect of the present disclosure, a method for correcting radial color distortion in an image captured by an imager is provided, the imager including a pixel sensor array and positioned behind a transflective coating to capture an image through the transflective coating, the image including a plurality of pixels corresponding to each pixel sensor in the pixel sensor array, the method being executed by a processor and comprising: receiving the image captured by the imager; determining correction data for each pixel based on a position of the corresponding pixel sensor within the pixel sensor array; and correcting a color value of each pixel of the captured image based on the correction data for the pixel to correct radial color distortion caused by the transflective coating.

[0005] Those skilled in the art will further understand and appreciate these and other features, advantages and objects of the present invention by referring to the following description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In each diagram:

[0007] Figure 1is a side cross-sectional view of a monitoring system according to an aspect of the present disclosure when implemented in a rearview mirror assembly;

[0008] Figure 2 is a side perspective view of the interior of a vehicle incorporating a monitoring system according to an aspect of the present disclosure;

[0009] Figure 3 is available for Figure 1 a cross-sectional view of an electro-optical mirror element of a rearview mirror assembly shown in;

[0010] Figure 4 It shows Figure 1 and 2 a circuit diagram in block form of the monitoring system shown in ; and

[0011] Figure 5 It is shown by Figure 1 、 2 and a flowchart of the method performed by the monitoring system shown in FIG. 4 . DETAILED DESCRIPTION

[0012] For the purpose of this description, the terms "upper", "lower", "right", "left", "rear", "front", "vertical", "horizontal" and their derivatives will be used in conjunction with Figure 1 . Unless otherwise specified, the term "front" shall refer to the mirror surface that is closer to the intended observer of the rearview assembly, and the term "rear" shall refer to the component surface that is farther from the intended observer of the rearview assembly. However, it is understood that the present invention may assume various alternative orientations, except where expressly specified to the contrary. It is also understood that the specific devices and processes illustrated in the drawings and described in the following specification are merely exemplary embodiments of the inventive concepts defined in the appended claims. Accordingly, unless the claims expressly state otherwise, specific dimensions and other physical characteristics relating to the embodiments disclosed herein should not be considered limiting.

[0013] The terms "including," "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements may include not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more constraints, the phrase "comprising" preceding an element does not preclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0014] Rearview mirror assemblies are known to include electro-optical mirror elements, such as electrochromic mirror elements. It is also known to use a transflective (i.e., partially reflective, partially transmissive) coating so that light can be transmitted from the display through the mirror element, or to a sensor or image sensor. This allows the mirror element to still function as a mirror by reflecting an image backward toward the driver, while also shielding electronic components behind the mirror, such as a display, image sensor, or other sensor.

[0015] When an image sensor for capturing images of the vehicle interior is placed behind a mirror element and its transflective coating, the light transmitted to the image sensor is affected by the transmissive properties of the transflective coating. Furthermore, it may be desirable for the image sensor to capture images using infrared (IR) radiation under both daytime and nighttime conditions. However, when an imager is employed behind a transflective coating, there are challenges in utilizing the transflective coating to provide both transmission of IR radiation and uniform transmission across the visible spectrum. In particular, applicants have discovered that such transflective coatings can introduce radial color distortion to images captured by the imager in the visible spectrum. This radial color distortion is similar to applying a separate lens vignetting effect to each color channel. The different attenuation across the various color channels manifests as a color shift. For example, such radial color distortion can cause an image to have a green tint in the center of the image, with a pink ring surrounding the green center, while the outer portions of the image appear white.

[0016] refer to Figure 1 、 3 4, reference numeral 10 generally designates a monitoring system for a vehicle 30, the monitoring system including at least one imager 12 including a pixel sensor array and positioned behind a transflective coating 70 to capture images through the transflective coating 70. The monitoring system 10 also includes a control system 100 having a processor 102 capable of determining correction data for each pixel based on the position of the pixel within the pixel array and correcting the color value of each pixel of the captured image based on the correction data for the pixel to correct radial color distortion caused by the transflective coating 70.

[0017] The processor 102 can correct the distortion by applying a white balancing operation whose coefficients are derived from the transmission properties of the transflective coating 70 and the ray angle (i.e., pixel location) of the incident light. The white balancing operation is applied as the final step in image processing (i.e., on the rendered full-color image).

[0018] The distortion is modeled as a white balance shift whose correction coefficient depends on the position in the imager's pixel sensor array, the transmission properties of the transflective coating 70, the spectrum of the illumination source, and the relative orientation of the glass plane of the rear of the mirror element 40 to the imager plane (as noted below, the transflective coating 70 can be provided by the mirror element 40).

[0019] According to a first approach, processor 102 can determine color correction data for each pixel by accessing a lookup table containing correction data for each corresponding pixel sensor in the pixel sensor array. Such a lookup table can be created through calibration, whereby images of a flat, white, well-illuminated scene are captured with and without the transflective coating 70. The ratio between each color channel of the image sample with and without the coating represents the radial color distortion of the coating 70 itself. The correction can then be applied to any subsequent capture of the scene illuminated by a source substantially similar to the source used for calibration by multiplying this "ratio image" by the uncorrected image.

[0020] An alternative second method for deriving corrections uses a characterized transmission profile of the transflective coating 70, knowledge of the lens and imager system, and the position in the imager's pixel sensor array. The transmission curve provides the attenuation of various wavelengths of light (and therefore the red, green, and blue channels) based on the angle of incidence of the incident ray onto the plane of the transflective coating. Knowledge of the lens and image sensor—i.e., the optical center and lens distortion parameters, as well as the relative angle of the transflective coating to the image sensor plane—provides information to calculate the angle of incidence based on the position in the pixel array. This information is then used to create a lookup table 110, where the entries for the correction coefficients are indexed by the angle of incidence. Alternatively, the lookup table can be indexed by radial distance from the optical center of the image pixel array, where the angle of incidence is converted to radial distance from the optical center during lookup table generation. Processor 102 can then implement this method by calculating the angle of incidence for each pixel position in the current image, looking up the correction coefficient in lookup table 110, and applying the correction by multiplying the red, green, and blue coefficients of the correction data by the red, green, and blue values ​​of the pixel.

[0021] While white balancing operations are known, their typical application is to correct for color distortions inherent to the characteristics of the imager and / or the color of the light source illuminating the scene. These methods do not take into account any spatial / structural dependencies, as the properties of these color distortions are uniform across the entire pixel array. In most cases, any additional material with optical properties placed in front of the imager (i.e., a glass cover) is specifically designed to not introduce color distortions, or at least to introduce spatially uniform distortions.

[0022] While previous white balancing operations did not account for spatially correlated color distortion, the monitoring system 10 allows for greater flexibility in the design of coatings because the introduced radial color distortion can be effectively corrected in the image processing performed by the processor 102. This greater flexibility should provide better performing coatings, at least in applications requiring simultaneous IR and visible light processing (i.e., in-cabin monitoring products for the automotive market).

[0023] The first method of determining the correction data has the advantage of simple derivation of the correction coefficients and simple lookup of the coefficients at run time, but at the expense of a memory-intensive implementation (even if the entire 3-channel image buffer can be used with compression techniques, the memory footprint can be quite large). The second method has the advantage of using a lower memory footprint; a 3xN lookup table, where N is typically 360, is many orders of magnitude smaller than the image buffer. However, this does introduce greater computational complexity for the lookup and is a more difficult method for deriving the correction coefficients from the transmission curve.

[0024] Now refer to Figure 1 and 2 The monitoring system 10 can be incorporated into various structures (e.g., the structure of a vehicle 30). For example, the imaging device 12 can be disposed in a rearview mirror assembly 32 for monitoring the position or characteristics (e.g., the driver's position) of an occupant 33 in a cabin 34 of the vehicle 30. The rearview mirror assembly 32 can include a mirror element 40 including a transflective coating 70. The mirror element 40 can be an electro-optical mirror element having an electro-optical medium 45 such that the mirror element 40 can be switched to change reflectivity.

[0025] The rearview mirror assembly 32 may include a housing 36 defining an opening 38, with a mirror element 40 positioned in the opening. The mirror element 40 includes a front surface 42 facing the occupant 33 and a rear surface 44 generally facing away from the occupant 33. An optional display device 41 may be located between the housing 36 and the mirror element 40 for displaying information to the driver. As disclosed herein, the electro-optical mirror element 40 may be a single-layer, single-phase component, a multi-layer component, or a multi-phase component. A further description of the electro-optical mirror element 40 is provided below.

[0026] The monitoring system 10 may also include an illumination source 18 configured to emit at least one of infrared (IR) or near-infrared (NIR) illumination. A transmitting optical element 46 may be fixed relative to the illumination source 18 to collimate and / or diffract the illumination. A receiving optical element 48 may be fixed relative to the imaging device 12 to focus light into the imaging device. The housing 36 may include a connection hub 52, and the connection hub 52 may be connected to a mounting member 54. The mounting member 54 is configured to be connected to the vehicle 30 (or other environment), and the housing 36 may be movable relative to the mounting member 54 to orient the mirror element 40 at various angles relative to the occupant 33 (or other environmental location) to obtain different views and / or orientations of the environment relative to the driver.

[0027] Filter 50B can be fixed relative to imaging device 12. Filter 50B can be configured to selectively attenuate the transmission of one or more specific wavelength spectra. For example, filter 50B can be configured as a dynamic filter, such as a filter incorporating a liquid crystal structure. In this way, filter 50B can selectively filter light from illumination source 18 and / or ambient light. Examples of such filters are disclosed in U.S. patent application Ser. No. 18 / 520,886, filed on November 28, 2023, entitled “MONITORING SYSTEM,” the entire disclosure of which is incorporated herein by reference.

[0028] refer to Figure 3 The electro-optical mirror element 40 may include a first substrate 60, a second substrate 62, a first electrode 64 disposed on a rear surface 61 of the first substrate 60, a second electrode 65 disposed on a front surface 63 of the second substrate 62, a seal 66 disposed between the substrates to define a sealed cavity 67 therebetween, and an electro-optic medium 45 disposed in the cavity 67 between the first electrode 64 and the second electrode 65. Figure 3 In the example shown in , the second electrode 65 can be a coating formed of one or more layers of conductive material that also have optical properties so as to form a transflective coating 70. In other words, the coatings 65, 70 serve as both the transflective coating 70 and the second electrode 65. It is also possible to form a separate transflective coating 70 that is located below a highly transmissive conductive layer that serves as the second electrode 65. Another possible configuration is to provide the transflective coating 70 on the rear surface 44 of the second substrate 62 while using a transparent electrode layer on the front surface 63 as the second electrode 65.

[0029] Now refer to Figure 4The control system 100 of the monitoring system 10 may include at least one processor 102. In some embodiments, the at least one processor 102 may include a combination of a processor associated with and controlling the imager 12 and a processor associated with and controlling the electro-optical element 40. However, in some embodiments, it should be understood that the processor 102 is a processor that issues global instructions to several (e.g., all or selected) components of the monitoring system 10. Each processor 102 may include a memory 106. The processor 102 may be any suitable processor. The memory 106 may include instructions that, when executed by the processor 102, cause the processor 102 to perform at least the functions associated with the components of the monitoring system 10. Thus, the imager 12, the illumination source 18, the electro-optical element 40, and the display 41 may be controlled by the control system 100 (e.g., the at least one processor 102 or a processor associated therewith). Therefore, the memory 106 may include software 108 and a lookup table 110.

[0030] Figure 5 An example of a method 200 performed by the processor 102 when receiving an image during normal operation is shown. The method 200 is generally described below and encompasses the first and second methods described above. First, the processor 102 controls the imager 12 to capture an image (step 202). Next, the processor 102 may determine correction data for each pixel based on the pixel's position within the pixel array of the imager 12 (step 204). The processor 102 may then correct the color value of each pixel of the captured image based on the correction data for that pixel to correct for radial color distortion caused by the transflective coating 70 (step 206). The processor 102 may repeat steps 202-206 until monitoring is no longer required. The correction data for each pixel may be stored in a lookup table 110 having correction data for each pixel in the pixel array according to the first method described above, or may be calculated from data contained in the lookup table 110 having a smaller amount of data stored therein according to the second method described above.

[0031] Although the monitoring system 10 is described as being located in the rearview mirror assembly 32, portions of the system or all of the system may be located elsewhere. Furthermore, although the system 10 is designed to correct for radial chromatic aberration caused by the transflective coating 70, the system may also correct for radial chromatic aberration caused by any other coating (whether in addition to or in place of the transflective coating 70) or by any other optical element placed in front of the imager 12.

[0032] Those skilled in the art will appreciate that the described invention and the construction of other components are not limited to any particular material. Unless otherwise described herein, other exemplary embodiments of the invention disclosed herein may be formed from a wide variety of materials.

[0033] For purposes of this disclosure, the term "couple" (in all its forms, coupling, coupling, coupled, etc.) generally means the joining of two components (electrical or mechanical) directly or indirectly to one another. Such joining may be stationary in nature or movable in nature. Such joining may be achieved using the two (electrical or mechanical) components and any additional intermediate members that are integrally formed as a single unitary body with one another or with the two components. Unless otherwise stated, such joining may be permanent in nature or removable or releasable in nature.

[0034] It is also worth noting that the construction and arrangement of the elements of the present invention as shown in the exemplary embodiments are merely illustrative. Although only a few embodiments of the present invention are described in detail in this disclosure, it will be readily apparent to those skilled in the art who review this disclosure that many modifications are possible (e.g., changes in the size, dimensions, structure, shape and proportion, parameter values, mounting arrangements, use of materials, colors, orientations, etc. of various elements) without substantially departing from the novel teachings and advantages of the stated subject matter. For example, an integrally formed element may be constructed from multiple elements, or an element shown as multiple elements may be integrally formed, the operation of the interface may be reversed or otherwise changed, the length or width of the structure and / or the components or connectors or other elements of the system may be changed, and the nature or number of adjustment positions between elements may be changed. It should be noted that the elements and / or assemblies of the system may be made of any of a wide variety of materials that provide sufficient strength or durability, and may be in any of a wide variety of colors, textures, and combinations. Therefore, all such modifications are intended to be included within the scope of this innovation. Other substitutions, modifications, changes, and omissions may be made in the design, operating conditions, and arrangement of desired and other exemplary embodiments without departing from the spirit of this innovation.

[0035] It should be understood that any described process or step within a described process can be combined with other disclosed processes or steps to form structures within the scope of the present invention.The exemplary structures and processes disclosed herein are for illustrative purposes and should not be construed as limiting.

[0036] It will be further understood that changes and modifications may be made in the structures and methods described above without departing from the inventive concepts and it will be further understood that such concepts are intended to be covered by the appended claims unless the language of the claims expressly states otherwise.

Claims

1. A rearview mirror assembly for a vehicle, comprising: an electro-optical element having at least one semi-transmissive and semi-reflective coating; at least one imager comprising a pixel sensor array and positioned behind the electro-optical element to capture an image through the transflective coating, the image comprising a plurality of pixels corresponding to each pixel sensor in the pixel sensor array; a processor capable of: receiving the image captured by the imager; determining correction data for each pixel based on a position of the corresponding pixel sensor within the pixel sensor array; and A color value of each pixel of the captured image is corrected based on the correction data for the pixel to correct radial color distortion caused by the transflective coating.

2. The rearview mirror assembly of claim 1 , wherein the processor determines the correction data for each pixel by accessing a lookup table.

3. The rearview mirror assembly of claim 2, wherein the lookup table has correction data for each pixel in the pixel sensor array.

4. The monitoring system of claim 3 , wherein the correction data stored in the lookup table is determined by calibration whereby image samples of a flat, white, well-illuminated scene are captured with and without the transflective coating in front of the imager, the ratio between each color channel of the image samples with and without the coating representing the radial color distortion of the transflective coating itself.

5. The rearview mirror assembly of claim 2, wherein the processor determines the correction data for each pixel by calculating an angle of incidence for each pixel location in the captured image and looking up a correction factor in the lookup table.

6. A rearview mirror assembly according to any one of claims 1 to 5, wherein the processor corrects the color value of each pixel of the captured image by multiplying the red coefficient, green coefficient and blue coefficient of the correction data by the red value, green value and blue value of the pixel.

7. A monitoring system for a vehicle, comprising: Semi-transmissive and semi-reflective coating; at least one imager comprising a pixel sensor array and positioned behind the transflective coating to capture an image through the transflective coating, the image comprising a plurality of pixels corresponding to each pixel sensor in the pixel sensor array; a processor capable of: receiving the image captured by the imager; determining correction data for each pixel based on a position of the corresponding pixel sensor within the pixel sensor array; and A color value of each pixel of the captured image is corrected based on the correction data for the pixel to correct radial color distortion caused by the transflective coating.

8. The monitoring system of claim 7, wherein the processor determines the correction data for each pixel by accessing a lookup table.

9. The monitoring system of claim 8, wherein the lookup table has correction data for each pixel in the pixel sensor array.

10. The monitoring system of claim 9 , wherein the correction data stored in the lookup table is determined by calibration whereby image samples of a flat, white, well-illuminated scene are captured with and without the transflective coating in front of the imager, the ratio between each color channel of the image samples with and without the coating representing the radial color distortion of the transflective coating itself.

11. The monitoring system of claim 8, wherein the processor determines the correction data for each pixel by calculating an angle of incidence for each pixel location in the captured image and looking up a correction factor in the lookup table.

12. The monitoring system according to any one of claims 7 to 11, wherein the processor corrects the color value of each pixel of the captured image by multiplying the red coefficient, green coefficient, and blue coefficient of the correction data by the red value, green value, and blue value of the pixel.

13. A monitoring system according to any one of claims 7 to 12, and further comprising a rearview assembly, the rearview assembly comprising an electro-optical element, wherein the transflective coating is part of the electro-optical element, and the image sensor is located in the rearview assembly.

14. The monitoring system of claim 13, wherein the processor is located in the rearview assembly.

15. A method of correcting radial color distortion in an image captured by an imager, the imager comprising an array of pixel sensors and positioned behind a transflective coating to capture an image through the transflective coating, the image comprising a plurality of pixels corresponding to each pixel sensor in the array of pixel sensors, the method being executed by a processor and comprising: receiving the image captured by the imager; determining correction data for each pixel based on a position of a corresponding pixel sensor within the pixel sensor array; as well as A color value of each pixel of the captured image is corrected based on the correction data for the pixel to correct radial color distortion caused by the transflective coating.

16. The method of claim 15, wherein the step of determining correction data comprises accessing a lookup table.

17. The method of claim 16, wherein the lookup table contains correction data for each corresponding pixel sensor in the pixel sensor array.

18. The method of claim 17 , wherein the correction data stored in the lookup table is determined by calibration whereby image samples of a flat, white, well-illuminated scene are captured with and without the transflective coating in front of the imager, the ratio between the image samples with and without the coating in each color channel being representative of the radial color distortion of the transflective coating itself.

19. The method of claim 16, wherein the lookup table contains correction data in the form of correction coefficients indexed by one of an angle of incidence and a radial distance from an optical center of the pixel sensor array.

20. The method of claim 16, wherein the step of correcting the color value of each pixel of the captured image comprises calculating the incident angle for each pixel position in the captured image, looking up the correction coefficient in the lookup table based on the incident angle, and correcting the color value of each pixel of the captured image by multiplying a red coefficient, a green coefficient, and a blue coefficient of the correction data by a red value, a green value, and a blue value of the pixel.

Images