Side view camera resolution testing and dirt residue detection and automatic cleaning / warning triggering

Abstract

A vehicle includes an imaging device that operates a camera. The imaging device includes a processor. The processor obtains a first image of the test object via the camera, determines a resolution of the first image, determines a contrast of the first image when the resolution is less than a resolution threshold, compares the contrast to a contrast threshold, activates the cleaning system to clean the camera when the contrast is less than the contrast threshold, obtains a second image of the test object, and transmit an alert signal to a driver of the vehicle based on the second image.

Description

BACKGROUND

[0001] The subject disclosure relates to vehicle-based imaging systems, and in particular, to a system and method for testing camera resolution by calculating image contrast and detecting the presence of debris on a camera of a vehicle-based imaging system, thereby enabling a cleaning process of the camera.

[0002] Vehicles can now be produced with imaging systems that replace side mirrors, the imaging systems including a camera coupled to a monitor. The camera is external to the vehicle and the monitor is internal to the vehicle at a convenient viewing location for the vehicle driver. The resolution of the camera is defined by its ability to resolve the smallest possible feature in an image. In order to obtain proper image resolution, the camera must work at a minimum resolution. Dust, water, and / or other debris can accumulate at the lens or glass surface of the camera, thereby affecting the contrast of the images captured by the camera. It is therefore desirable to provide a system and method for cleaning debris from the camera. SUMMARY

[0003] In one example embodiment, a method of operating a camera of an imaging device of a vehicle is disclosed. The method includes obtaining, via the camera, a first image of a test object, the camera including an image sensor having an array of pixels, determining a resolution of the first image, comparing the resolution to a resolution threshold, determining, at a processor, a contrast of the first image when the resolution is less than the resolution threshold, wherein the contrast is based on a difference between a first gray scale value of a first pixel of the first image having a maximum intensity and a second gray scale value of a second pixel of the first image having a minimum intensity, comparing, at the processor, the contrast to a contrast threshold, performing, via the processor, a cleaning operation to clean the camera when the contrast is less than the contrast threshold, obtaining a second image of the test object, and sending an alert signal to a driver of the vehicle based on the second image.

[0004] In addition to one or more of the features described herein, the method includes segmenting the first image into at least a first region and a second region, determining a first contrast of the first region and a second contrast of the second region, and performing the cleaning operation when at least one of the first contrast and the second contrast is less than the contrast threshold.

[0005] In addition to one or more of the features described herein, the method includes determining the resolution of the first image using a modulation transfer function.

[0006] In addition to one or more of the features described herein, the test object is at least one of a door handle of the vehicle, a taillight of the vehicle, a test pattern printed on the vehicle within a field of view of the camera, and a set of LEDs on the vehicle and within the field of view of the camera.

[0007] In addition to one or more of the features described herein, the method further includes performing a cleaning operation at a subsequent time when the contrast of the second image is less than a contrast threshold.

[0008] In addition to one or more of the features described herein, the alert signal is at least one of a visual signal and an audio signal.

[0009] In addition to one or more of the features described herein, the method further includes sending an alert signal when at least one of a resolution of the second image is less than a resolution threshold and a contrast of the second image is less than a contrast threshold.

[0010] In another example embodiment, an imaging device for a vehicle is disclosed. The imaging device includes a camera including a pixel array, a cleaning system, and a processor. The processor is configured to obtain a first image of a test object via the camera, determine a resolution of the first image, compare the resolution to a resolution threshold, determine a contrast of the first image when the resolution is less than the resolution threshold, wherein the contrast is based on a difference between a first grayscale value of a first pixel of the first image having a maximum intensity and a second grayscale value of a second pixel of the first image having a minimum intensity, compare the contrast to a contrast threshold, activate the cleaning system to clean the camera when the contrast is less than the contrast threshold, obtain a second image of the test object, and send an alert signal to a driver of the vehicle based on the second image.

[0011] In addition to one or more of the features described herein, the processor is further configured to segment the first image into at least a first region and a second region, determine a first contrast of the first region and a second contrast of the second region, and activate the cleaning system when at least one of the first contrast and the second contrast is less than a contrast threshold.

[0012] In addition to one or more of the features described herein, the processor is further configured to determine the resolution of the first image using a modulation transfer function.

[0013] In addition to one or more of the features described herein, the test object is at least one of a door handle of the vehicle, a taillight of the vehicle, a test pattern printed on the vehicle within a field of view of the camera, and a set of LEDs on the vehicle and within the field of view of the camera.

[0014] In addition to one or more of the features described herein, the processor is further configured to activate the cleaning system at a subsequent time when the contrast of the second image is less than a contrast threshold.

[0015] In addition to one or more of the features described herein, the imaging device further includes an interface to provide the alert signal to the driver as at least one of a visual signal and an audio signal.

[0016] In addition to one or more of the features described herein, the camera is located on a side of the vehicle.

[0017] In yet another example embodiment, a vehicle is disclosed. The vehicle includes a test object located on the vehicle and an imaging system. The imaging system includes a camera, a cleaning system, and a processor, the camera including an image sensor having a pixel array, the cleaning system to clean a glass surface of the camera. The processor is configured to obtain, via the camera, a first image of the test object, determine a resolution of the first image, compare the resolution to a resolution threshold, determine a contrast of the first image when the resolution is less than the resolution threshold, wherein the contrast is based on a difference between a first gray scale value of a first pixel of the first image having a maximum intensity and a second gray scale value of a second pixel of the first image having a minimum intensity, compare the contrast to a contrast threshold, activate the cleaning system to clean the camera when the contrast is less than the contrast threshold, obtain a second image of the test object, and send an alert signal to a driver of the vehicle based on the second image.

[0018] In addition to one or more of the features described herein, the processor is further configured to segment the first image into at least a first region and a second region, determine a first contrast of the first region and a second contrast of the second region, and activate the cleaning system when at least one of the first contrast and the second contrast is less than a contrast threshold.

[0019] In addition to one or more of the features described herein, the processor is further configured to determine the resolution of the first image using a modulation transfer function.

[0020] In addition to one or more of the features described herein, the test object is at least one of a door handle of the vehicle, a taillight of the vehicle, a test pattern printed on the vehicle within a field of view of the camera, and a set of LEDs on the vehicle and within the field of view of the camera.

[0021] In addition to one or more of the features described herein, the processor is further configured to activate the cleaning system at a subsequent time when the contrast of the second image is less than the contrast threshold.

[0022] In addition to one or more of the features described herein, the vehicle further includes an interface to provide the alert signal to the driver as at least one of a visual signal and an audio signal.

[0023] The above features and advantages of the present disclosure, as well as other features and advantages, are readily apparent from the following detailed description, when taken in connection with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] Other features, aspects, and details are described below with reference to the drawings. The foregoing summary is not un BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A plan view of a vehicle according to an exemplary embodiment is shown;

[0026] Figure 2 A perspective view of the vehicle is shown from the driver's side.

[0027] Figure 3 This is a flowchart of a method for operating a camera cleaning system in an embodiment;

[0028] Figure 4 It is an image of a distant vehicle with headlights pointing towards the camera;

[0029] Figure 5 The graphs show various resolution levels for two points;

[0030] Figure 6 A flowchart illustrating a method for determining the resolution of an image in an illustrative embodiment is shown;

[0031] Figure 7 A flowchart illustrating a method for cleaning a camera of an imaging device based on image contrast is shown in an embodiment;

[0032] Figure 8 A flowchart illustrating a method for determining the endpoint of a cleaning operation in an embodiment is shown;

[0033] Figure 9 An illustrative image of a test pattern with black (low intensity) and white (high intensity) portions is shown;

[0034] Figure 10 A second image of the test pattern, taken by a camera, is shown, which experiences reduced image contrast due to slight water buildup.

[0035] Figure 11 A third image of the test pattern, captured by a camera, is shown, exhibiting reduced image contrast due to the significant accumulation of water; and

[0036] Figure 12 A fourth image of the test pattern, taken by a camera, is shown, showing reduced image contrast due to the accumulation of dirt. Detailed Implementation

[0037] The following description is exemplary in nature only and is not intended to limit this disclosure, its application, or use. It should be understood that throughout the drawings, corresponding reference numerals denote the same or corresponding parts and features.

[0038] According to an exemplary embodiment, Figure 1A plan view 100 of a vehicle 102 is shown. The vehicle 102 includes an imaging system 104. The imaging system 104 may be a side-view imaging system replacing one or more side mirrors of the vehicle. For illustrative purposes, the imaging system 104 includes a first imaging device 106a located on the driver's side of the vehicle and a second imaging device 106b located on the passenger's side of the vehicle. The first imaging device 106a includes a first camera 108a, a first processor 110a, a first monitor 112a, and a first cleaning system 114a. Other embodiments of the imaging system 104 may include imaging devices having cameras oriented in other possible directions, replacing the first imaging device 106a and the second imaging device 106b, or other than the first imaging device 106a and the second imaging device 106b.

[0039] A first camera 108a may be mounted on the outer surface of the driver's side door of vehicle 102. A first monitor 112a is located inside the vehicle, such as on the inner surface of the driver's side door. The first monitor 112a may be a flat panel display, such as a light-emitting diode (LED) display, and directs the images captured by the first camera 108a toward the driver 120 or occupant of vehicle 102. A first processor 110a is coupled to the first camera 108a and the first monitor 112a. Images captured by the first camera 108a are sent to the first processor 110a, which then sends the images to the first monitor 112a. The first processor 110a may also perform calculations using the images to determine camera resolution and / or image contrast. The first processor 110a may activate a first cleaning system 114a based on the contrast to clean the first camera 108a.

[0040] The first camera 108a can be a digital camera with an image sensor having a pixel array. Each pixel can output a voltage corresponding to the intensity of light received at the pixel or a grayscale value recorded at the pixel. Therefore, when a highly reflective portion of the image (such as a white portion of the image) falls on the pixel, the pixel can output a first grayscale value, and when a highly non-reflective portion of the image (such as a black portion of the image) falls on the pixel, the pixel can output a second grayscale value.

[0041] The first camera 108a may have a lens or glass cover for protecting the first camera 108a. The lens or glass cover accumulates water, dirt, or other debris that affects the contrast of the image captured by the first camera 108a. The first cleaning system 114a may use various methods, including jetting air, jetting fluid, etc., to clean the lens or glass cover. In various embodiments, the first cleaning system 114a may be an integrated component of the first camera.

[0042] Similarly, the second imaging device 106b can operate in the same manner as described herein with respect to the first imaging device 106a. The second imaging device 106b includes a second camera 108b, a second processor 110b, a second monitor 112b, and a second cleaning system 114b.

[0043] In various embodiments, the first processor 110a and the second processor 110b may be a single processor communicating with their respective cameras. The processors may be included in a controller for the vehicle. The controller may include processing circuitry, which may include application-specific integrated circuits (ASICs), electronic circuitry, a processor (shared, dedicated, or grouped) and memory executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components providing the described functionality. The controller may include a non-transitory computer-readable medium storing instructions that, when processed by one or more processors of the controller, implement methods for determining the resolution and contrast of an image according to one or more embodiments detailed herein, activating a cleaning system or cleaning operation based on the contrast to clean the camera, and evaluating new images once the cleaning system has cleaned the camera.

[0044] Vehicle 102 may also include an interface 118 for providing information to the driver. In various embodiments, interface 118 may be a head-up display, a central stack display, or a digital look-back mirror display. Information may be provided as visual signals, visual messages, visual images, etc. Interface 118 may also be a speaker (integrated into or separate from the display) with audio capabilities to transmit noise or audio signals. The information may include the results of cleaning and testing operations.

[0045] Figure 2 A perspective view 200 of a vehicle 102 at the driver's side is shown. Perspective view 200 shows a first camera 108a and several possible test objects within the field of view of the first camera. The first camera 108a can capture images of the objects and can determine the resolution and contrast of the images. Exemplary test objects include a door handle 202, a taillight 204, and a test pattern 206 on the side of the vehicle. In one embodiment, the test pattern 206 may be a printed test pattern. In another embodiment, the test pattern 206 may be a set of LEDs that can be illuminated for testing purposes.

[0046] like Figure 2 As shown, test pattern 206 includes a set of horizontal and vertical stripes. The stripes provide an image including high-intensity and low-intensity portions at the first camera.

[0047] Figure 3This is a flowchart 300 of a method for operating a cleaning system for a camera, as described in the embodiment. In block 302, an image is acquired by the camera. In block 304, the image resolution of the image is determined. In block 306, the image resolution is compared with a resolution threshold. If the image resolution is greater than the resolution threshold (i.e., an acceptable image resolution), the method proceeds to block 312. In block 312, the method ends. Returning to block 306, if the image resolution is less than or equal to the resolution threshold, the method proceeds to block 308.

[0048] In box 308, the image contrast is determined. In box 310, the contrast is compared to a contrast threshold. If the contrast is greater than the contrast threshold (i.e., acceptable contrast), the method proceeds to box 312, where the method terminates. Returning to box 310, if the contrast is less than or equal to the contrast threshold, the method proceeds to box 314. In box 314, a cleaning signal is sent to the cleaning system to activate it. From box 314, the method can return to box 302 to obtain another image, allowing the results of the cleaning operation to be tested.

[0049] Figure 4 This is an image 400 of a distant vehicle with headlights pointing towards the camera. The headlights cause glare in the image, resulting in image saturation. Glare may be due to surface defects, scratches, contamination, or fogging. The camera's resolution should be sufficient to distinguish the headlights as two separate headlights rather than a single headlight.

[0050] Figure 5 A graph 500 shows curves at various resolution levels for two points. The incoherent image intensity is the convolution of the square of the point spread function and the object intensity. A first point spread function 502 and a second point spread function 504 are shown. For example, the first point spread function 502 could be one of the headlights in image 400, and the second point spread function 504 could be another headlight. If the maximum value of the point spread function of the first target point falls within the minimum value of the point spread function of the second target point, then the first target point and the second target point are considered to be distinguishable.

[0051] Graph 508 illustrates the indistinguishable case, where the superimposed envelope 506 of the first point spread function 502 and the second point spread function 504 shows only a single peak. Graph 510 illustrates the just-distinguishable case, where the envelope has two peaks, with the maximum value of the first point spread function 502 falling into the first minimum value of the second point spread function 504, and vice versa. Graph 512 illustrates the distinguishable case. The resolution of an image can be determined using a modulation transfer function, which can be a slanted-edge modulation transfer function as disclosed herein.

[0052] Figure 6A flowchart 600 illustrates a method for determining the resolution of an image in an illustrative embodiment. In block 602, an image is received from a camera at a processor. In block 604, the image is corrected for distortions attributable to factors such as the curvature of the lens (i.e., a wide-angle lens). In block 606, the image is converted to grayscale levels. In block 608, an edge spread function is created by projecting a two-dimensional array of values ​​from the image onto a one-dimensional array.

[0053] In box 610, the edge spread function is convolved with a finite impulse response filter to obtain the line spread function. In box 612, a smoothing function is applied to the line spread function. In box 614, a discrete Fourier transform is performed on the line spread function to obtain its frequency domain representation. In box 616, the amplitude of the frequency domain representation is plotted to obtain the slanted edge modulation transfer function. The slanted edge modulation transfer function is used to determine the image resolution. The resolution quality of the camera system is determined by comparing the resolution to a resolution threshold. If the resolution is less than the resolution threshold, the system can issue an alert to the driver and / or continue to check the image contrast.

[0054] Figure 7 A flowchart 700 illustrating a method for using a camera in an image-based contrast-cleaning imaging apparatus, as shown in an embodiment, is illustrated. In block 702, an image is received from the camera. The image may include a test object. In block 704, the intensity of each pixel in the image is measured, and the maximum intensity (e.g., the whitest pixel) and the minimum intensity (e.g., the blackest pixel) are identified. The maximum intensity may be represented by a first grayscale value output by a first pixel having the highest intensity, or it may be the average of the grayscale values ​​of a pre-selected number of pixels having the highest intensity. Similarly, the minimum intensity may be represented by a second grayscale value output by a second pixel having the lowest intensity, or it may be the average of the grayscale values ​​of a pre-selected number of pixels having the lowest intensity.

[0055] In box 706, the contrast ratio C is calculated using the maximum and minimum gray values, as shown in equation (1):

[0056] C = (max - min) / (max + min) Equation (1)

[0057] Where max is the grayscale value of the pixel with the highest intensity, and min is the grayscale value of the pixel with the lowest intensity. In box 708, the contrast is compared to a contrast threshold. If the contrast is greater than the contrast threshold (i.e., indicating the camera's acceptable contrast level), the method proceeds to box 710. In box 710, the method terminates.

[0058] Returning to box 708, if the contrast is less than or equal to the contrast threshold, the method proceeds to box 712. In box 712, the cleaning system is activated to clean the camera (e.g., clean the glass cover).

[0059] Figure 8 A flowchart 800 illustrating a method for determining the endpoint of a cleaning operation in an embodiment is shown. The method includes activating the cleaning system a selected number of times until the image contrast increases above a contrast threshold, or until the number of times the cleaning system is activated exceeds a count threshold. In block 802, a counter is initialized to zero. In block 804, an image (first image) is acquired at a first camera, and the image contrast is determined or calculated. In block 806, the counter is incremented by 1. In block 808, the contrast is compared to a contrast threshold to determine whether the camera's cleaning system is activated. If the contrast is greater than or equal to the contrast threshold (i.e., cleaning is not required), the method proceeds to block 810. In block 810, the cleaning operation is stopped.

[0060] Returning to box 808, if the contrast is less than a contrast threshold (i.e., cleaning is required), the method proceeds to box 812. In box 812, a counter is compared to a counting threshold. If the counter is equal to or greater than the counting threshold, the method proceeds to box 814. In box 814, the processor sends an alarm signal to the driver. The alarm signal may take the form of a message displayed on a monitor or an audible signal. The cleaning operation is then stopped.

[0061] Returning to box 812, if the counter is less than a counting threshold, the method proceeds to box 816. In box 816, the cleaning system is activated to clean the camera. The method returns from box 816 to box 804, where a second or subsequent image is captured. In box 804, the contrast of the subsequent image can be obtained. Alternatively, the resolution can be checked in box 804, and if the resolution is unacceptable (i.e., less than a resolution threshold), the contrast can be obtained. The method then proceeds to clean the camera at a second or subsequent time (box 816) or terminates the process based on the second or subsequent image (box 810 or box 814).

[0062] For the purpose of explanation, Figures 9-11 Images with different contrast levels are shown. Figure 9 An illustrative image 900 shows a test pattern with black (low intensity) and white (high intensity) portions. The illustrative image 900 has 100% contrast. Figure 10 A second image 1000 of the test pattern, captured by a camera, is shown, exhibiting reduced image contrast due to slight water accumulation. The second image 1000 has a contrast ratio of 80%. Figure 11A third image 1100 of the test pattern, captured by a camera, is shown, exhibiting reduced image contrast due to the significant accumulation of water. The third image 1100 has a contrast ratio of 45%. Figure 12 A fourth image 1200 of the test pattern, captured by a camera, is shown, exhibiting reduced image contrast due to the accumulation of dirt. The fourth image 1200 has a contrast ratio of 95%.

[0063] The fourth image 1200 also illustrates a contrast test using image segmentation. For illustrative purposes, the fourth image 1200 is segmented into two separate regions, and the contrast is calculated for each region. However, in other embodiments, the image can be segmented into any number of regions. The first region 1202 is free of debris, while the second region 1204 has accumulated dirt. The contrast is determined for each of the first region 1202 and the second region 1204, respectively. The first region 1202 has a first contrast of 100%, while the second region has a second contrast of 90%. The total contrast of the image is the average of the first and second contrasts, approximately 95%. When a contrast threshold is set, for example, 92%, the contrast of the entire image (i.e., 95%) is considered acceptable. However, when at least one of the first and second contrasts is less than the contrast threshold, a cleaning system is activated. In the fourth image 1200, the second contrast (i.e., 90%) is less than this contrast threshold (i.e., 92%). Therefore, the cleaning system is activated.

[0064] The terms “an” and “a” do not indicate a limitation of quantity, but rather that at least one of the referenced items is present. Unless the context clearly indicates otherwise, the term “or” means “and / or”. Throughout the specification, reference to “aspect” means that a particular element described in connection with that aspect (e.g., a feature, structure, step, or characteristic) is included in at least one aspect described herein and may or may not be present in other aspects. Furthermore, it should be understood that the described elements may be combined in any suitable manner in the aspects.

[0065] When an element, such as a layer, film, region, or substrate, is referred to as being “on” another element, it can be directly on the other element, or there may be intermediate elements present. Conversely, when an element is referred to as being “directly” on another element, there are no intermediate elements present.

[0066] Unless otherwise stated herein, all test standards are the most recent standards in effect as of the date of filing of this application, or, if priority is claimed, the date of filing of the earliest priority application in which a test standard appears.

[0067] Unless otherwise defined, the technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0068] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and elements can be substituted with equivalents without departing from its scope. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the basic scope of this disclosure. Therefore, it is intended that this disclosure be limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.

Claims

1. A method of operating a camera of an imaging device of a vehicle, comprising: obtaining, via the camera, a first image of a test object, the camera comprising an image sensor having a pixel array; determining a resolution of the first image; comparing the resolution to a resolution threshold; determining, at a processor, a contrast of the first image when the resolution is less than the resolution threshold, wherein the contrast is based on a difference between a first grayscale value of a first pixel of the first image having a maximum intensity and a second grayscale value of a second pixel of the first image having a minimum intensity; comparing, at the processor, the contrast to a contrast threshold; performing, via the processor, a cleaning operation to clean the camera when the contrast is less than the contrast threshold; obtaining a second image of the test object; and sending an alert signal to a driver of the vehicle based on the second image.

2. The method of claim 1, further comprising segmenting the first image into at least a first region and a second region, determining a first contrast of the first region and a second contrast of the second region, and performing the cleaning operation when at least one of the first contrast and the second contrast is less than the contrast threshold.

3. The method of claim 1, further comprising determining the resolution of the first image using a modulation transfer function. the test object is at least one of: (i) a door handle of the vehicle; (ii) a taillight of the vehicle; (iii) a test pattern printed on the vehicle within a field of view of the camera; and (iv) a set of LEDs on the vehicle and within the field of view of the camera.

4. The method of claim 1, wherein, 5. The method of claim 1, further comprising performing the cleaning operation at a subsequent time when a contrast of the second image is less than the contrast threshold.

6. The method of claim 1, further comprising sending the alert signal when at least one of: (i) a resolution of the second image is less than the resolution threshold; and (ii) a contrast of the second image is less than the contrast threshold.

7. An imaging device for a vehicle, comprising: a camera comprising a pixel array; a cleaning system; a processor configured to: obtain, via the camera, a first image of a test object; determine a resolution of the first image; compare the resolution to a resolution threshold; determine, at a processor, a contrast of the first image when the resolution is less than the resolution threshold, wherein the contrast is based on a difference between a first grayscale value of a first pixel of the first image having a maximum intensity and a second grayscale value of a second pixel of the first image having a minimum intensity; compare, at the processor, the contrast to a contrast threshold; activate the cleaning system to clean the camera when the contrast is less than the contrast threshold; obtain a second image of the test object; and send an alert signal to a driver of the vehicle based on the second image. ​ ​ 8. The imaging device of claim 7, wherein the processor is further configured to segment the first image into at least a first region and a second region, determine a first contrast of the first region and a second contrast of the second region, and activate the cleaning system when at least one of the first contrast and the second contrast is less than the contrast threshold.

9. The imaging device of claim 7, wherein the test object is at least one of: (i) a door handle of the vehicle; (ii) a taillight of the vehicle; (iii) a test pattern printed on the vehicle within a field of view of the camera; and (iv) a set of LEDs on the vehicle and within a field of view of the camera.

10. The imaging device of claim 7, wherein the processor is further configured to activate the cleaning system at a subsequent time when a contrast of the second image is less than the contrast threshold.

Images