Automatic infrared LED control for camera monitor systems

Abstract

To provide a camera monitor system and a method for improving driver visibility in a low light situation in a camera monitor system (CMS) using an infrared LED for night vision.SOLUTION: A camera monitor system CMS 30 includes a camera arm 16 that has a camera 20 with an image capture unit 36 that is configured to capture an image of a desired field of view. A display 18 is configured to display the desired field of view. Infrared (IR) LEDs 38 and 40 illuminate IR LED illumination areas 26 and 28 that are at least portions of the desired field of view. A controller is capable of communicating with the image capture unit and the IR LEDs, and selects at least a first region of interest (ROI) and a second ROI from the captured image. The first ROI is indicative of an amount of ambient light. The controller adjusts the IR LED state based on first ROI luminance and / or second ROI luminance.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a CMS system that uses infrared LEDs for night vision. [Background technology]

[0002] The use of infrared LEDs can improve driver visibility in low light conditions. The infrared LEDs are controlled so they are not continuously on to manage heat. When using a camera monitor system with night vision, the image displayed to the driver inside the vehicle can become oversaturated, making it difficult to see objects or people clearly. Summary of the Invention

[0003] In one exemplary embodiment, a camera monitor system includes a camera arm having a camera with an image capture unit configured to capture an image of a desired field of view. The system further includes a display configured to display the desired field of view, an infrared LED configured to illuminate at least a portion of the desired field of view, and a controller in communication with the image capture unit and the infrared LED. The controller is configured to select at least first and second regions of interest (ROIs) from the captured image. The first ROI indicates an amount of ambient light. The controller is configured to determine a luminance of each of the first and second ROIs. The controller is configured to adjust an infrared LED state of the infrared LEDs based on the luminance of the first ROI, and the controller is configured to adjust the infrared LED state based on the luminance of the second ROI.

[0004] In a further embodiment of any of the above, the first ROI and the second ROI correspond to different first and second portions of the same image, the first ROI being above the horizon and in the sky in the desired field of view, and the second ROI being behind along a vehicle having a camera monitoring system.

[0005] In a further embodiment of any of the above, the image capture unit is configured to capture an RGB image, and the controller is configured to convert the RGB images of the first and second ROIs to HSV.

[0006] In a further embodiment of any of the above, the controller is configured to change the infrared LED from an off state to an on state, or from the on state to an off state.

[0007] In further embodiments of any of the above, the controller is configured to vary the amount of infrared LED illumination other than a fully on infrared LED state or a fully off infrared LED state.

[0008] In a further embodiment of any of the above, the controller is configured to adjust the infrared LED state based on a vehicle operating condition and includes a switch having an automatic position and a manual position, the controller is configured to adjust the infrared LED state in response to the switch being in the automatic position.

[0009] In another exemplary embodiment, a method for automatically controlling a vehicle night vision system includes capturing an image in a desired field of view; selecting at least first and second regions of interest (ROIs) from the captured image, the first ROI indicating an amount of ambient light; determining a luminance of each of the first and second ROIs, the first ROI indicating an amount of ambient light; adjusting an infrared LED state of infrared LEDs based on the luminance of the first ROI; and adjusting the infrared LED state based on the luminance of the second ROI.

[0010] In a further embodiment of any of the above, the first and second ROIs correspond to different first and second portions of the same captured image.

[0011] In a further embodiment of any of the above, the first ROI is above the horizon and in the sky in the desired field of view, and the second ROI is behind along the vehicle having the night vision system.

[0012] In a further embodiment of any of the above, the desired field of view includes both Class II and Class IV views.

[0013] In a further embodiment of any of the above, the step of capturing an image includes capturing an RGB image, and the step of determining the luminance includes converting the RGB images of the first and second ROIs to HSV (hue, saturation, value).

[0014] In a further embodiment of any of the above, the step of determining the luminance comprises calculating a median brightness value.

[0015] In a further embodiment of any of the above, the step of determining the luminance includes applying a low pass filter to the brightness for each of the first and second ROIs.

[0016] In a further embodiment of any of the above, the low pass filters for each of the first and second ROIs are different from each other.

[0017] In a further embodiment of any of the above, the determining the brightness step includes comparing the HSV brightness to a desired brightness, and the adjusting the infrared LED state step includes changing the infrared LED state only if the brightness exceeds an offset from the desired brightness.

[0018] In a further embodiment of any of the above, adjusting the infrared LED state includes automatically turning the infrared LED on or off.

[0019] In a further embodiment of any of the above, adjusting the infrared LED state includes turning the infrared LED from an off state to an on state or from an on state to an off state.

[0020] In a further embodiment of any of the above, adjusting the infrared LED state includes varying an amount of infrared LED illumination other than a fully on or fully off state of the infrared LED.

[0021] In a further embodiment of any of the above, adjusting the infrared LED state is performed based on an operating state of the vehicle.

[0022] In a further embodiment of any of the above, the method includes a switch having an automatic position and a manual position, and the step of adjusting the infrared LED state is performed in response to the switch being in the automatic position.

[0023] The present disclosure may be better understood by reference to the following detailed description when considered in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0024] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera monitor system (CMS) used to provide at least Class II and Class IV views. [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera monitor system providing Class II, Class IV, Class V, and Class VI views. [Figure 2] 1 is a schematic diagram of a CMS with night vision capabilities in accordance with the systems and methods of the present disclosure. [Figure 3] 3A and 3B show views displayed during daytime and nighttime, respectively, using an image capture unit with first and second regions of interest. [Figure 4]10 is an example of how information from first and second regions of interest can be used to control an infrared LED. DETAILED DESCRIPTION OF THE INVENTION

[0025] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. Like reference numbers and symbols in the various drawings indicate like elements.

[0026] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be of any configuration. While a commercial truck is discussed in this disclosure, the disclosed system may be applied to other types of vehicles. The vehicle 10 incorporates a camera monitor system (CMS) 30 (FIG. 2) that includes driver and passenger side camera arms 16a, 16b (generally "16") mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 30 may also be used to completely replace side view mirrors. In additional examples, multiple camera arms 16 may be included on each side, with each arm 16 housing one or more cameras and / or mirrors.

[0027] Each camera arm 16a, 16b includes a base 32 fixed to, for example, the cab 12, as shown in FIG. 2. A pivoting arm 34 is supported by the base 34 and may be articulated relative to the base either manually or using a powered retraction mechanism. Returning to FIG. 1B, at least one rear-facing camera 20a, 20b (generally "20") is disposed within each camera arm. The exterior cameras 20a, 20b each provide an exterior field of view (FOVEX1, FOVEX2), each including at least one of a Class II and a Class IV view (FIG. 1B), which are legally defined views in the commercial trucking industry. The Class II view for a given side of the vehicle 10 is a subset of the Class IV view for the same side of the vehicle 10. In one example, both the Class II and Class IV views are provided by a single camera providing a wide-angle view. If desired, multiple cameras may be used on each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in the European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. References to "class" views are not intended to be limiting, but rather as an example of the type of view provided to the display by a particular camera. Each arm 16 a, 16 b may also provide a housing that encloses electronics configured to provide various features of the CMS 30.

[0028] First and second video displays 18a, 18b (generally "18") are positioned within the vehicle cab 12 on or near the A-pillars on the driver's and passenger's sides, respectively, to display Class II and Class IV views on each side of the vehicle 10, thereby providing rearward along-the-vehicle views captured by exterior cameras 20a, 20b. The Class II and Class IV views may be provided by cropping portions of the image from wide-angle cameras.

[0029] If Class V and Class VI view footage is also desired, camera housing 16c and camera 20c may be positioned at or near the front of vehicle 10 to provide these views (FIG. 1B). A third display 18c located within cab 12 near the top center of the windshield can be used to display Class V and Class VI views toward the driver of vehicle 10. Displays 18a, 18b, 18c (generally, displays 18) face toward a driver area 24 within cab 22, where the driver is seated in driver's seat 26. The location, size, and field of view(s) streamed to a particular display may vary from the configurations described herein and still encompass the invention of this disclosure.

[0030] If Class VIII view video is desired, camera housings may be positioned on the sides and rear of vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of vehicle 10. In such an example, third display 18c may include one or more frames that display the Class VIII view. Alternatively, additional displays may be added near first, second, and third displays 18a, 18b, 18c to provide dedicated displays that provide the Class VIII view.

[0031] It should be understood that more or fewer displays than those generally described may be used, and that display images from multiple cameras may be combined into one display, or that images from particular fields of view may be provided on separate displays separate from other images.

[0032] The area behind a trailer is a common blind spot for all vehicles, but particularly for commercial trucks. Therefore, it is desirable to use a sensor, such as camera 20d, as shown in FIG. 1B, to provide the driver with some awareness of unseen objects at the rear of the trailer. A challenge with using a camera at the rear of a trailer is the long cab wiring used to transmit the video signal to a display in the cab. Dedicated wiring would add significant cost to the system. Furthermore, the images must be transmitted with minimal or no delay so that the objects can be displayed in real time.

[0033] FIG. 2 illustrates a night vision system 30. The system 30 includes an image capture unit 36 ​​configured to capture a desired field of view for the camera 20. The camera 20 is disposed on a pivot 34 of a camera arm 16 that is articulated relative to a fixed base portion 32 secured to the vehicle cab 12. The camera arm 16 also includes one or more infrared LEDs 38, 40. The infrared LEDs may be a single infrared LED or an array of multiple infrared LEDs, which may be controlled collectively or independently of one another. In one example, a first infrared LED array 38 provides one night vision illumination area 28, and a second array 40 provides another infrared LED illumination area 26, which may be illuminated on the side of the vehicle 10, as shown in FIG. 1B. It should be understood that a single infrared LED or an array of infrared LEDs may be used instead of the two arrays shown.

[0034] 2, a controller 42, which may be located within the vehicle cab 12, is in communication with the image capture unit 36 ​​and the infrared LEDs 38, 40. The controller 42 may include video processing to display images captured from the image capture unit 36 ​​in a desired format on the display 18, such as Class II and Class IV views.

[0035] The controller 42 communicates with various input and output devices. The controller 42 may receive information regarding the vehicle's operating state from vehicle components via a CAN bus. For example, a transmission gear position switch 48 may indicate whether the vehicle is in forward or reverse gear. A vehicle speed sensor 50 provides vehicle speed information. Another example of an input device is a switch movable between OFF, AUTOMATIC, and MANUAL. In the OFF position, the IR LED switches to an OFF IR LED state and is inoperative. In the MANUAL position, the driver can switch the IR LED to an ON IR LED state, regardless of whether the IR LED is turned on in automatic operating mode. In the AUTOMATIC position, the controller 42 uses an algorithm 60, which is described in more detail below and summarized in the manner shown in FIG. 4.

[0036] 3A and 3B, the display 18 includes at least two regions of interest (ROIs) shown as cropped portions of the same image provided by the image capture unit 36. These regions are not displayed on the display, but are instead 3A and 3B illustrate where regions may be located to provide the described functionality. More than two ROIs may be used if desired. The controller 42 is configured to determine the average luminance of each of the first and second ROIs 56, 58. The luminance values ​​may be normalized between 0 and 1, where 0 is pitch black and 1 is full light. An intermediate value of 0.5 may be used, for example, as a dividing point between daytime and nighttime. The 0.5 value may also be used to determine whether an oversaturated condition may occur at night, e.g., whether the luminance in the second ROI 58 is less than 0.5 at night. Other values ​​may be used if desired.

[0037] The first and second ROIs 56, 58 are each above the horizon and in the sky, providing a portion of the image along the vehicle, e.g., along the trailer 14, and rearward. The first ROI 56 is intended to provide an indication of the amount of ambient light, i.e., daytime or nighttime. In the illustrated example, the average brightness of the portion of the captured image within the first ROI 56 is 0.77, indicating daytime (FIG. 3A), while a value of 0.23 indicates nighttime (FIG. 3B). The second ROI 58 is intended to capture illumination that may saturate any image or object in the field of view when the infrared LEDs are turned on, e.g., by a vehicle with its headlights overtaking a tractor / trailer.

[0038] When the switch 46 is in the automatic position, the controller 42 is configured to adjust the infrared LED state of the infrared LEDs 38 and / or 40 based on the brightness of the first ROI. That is, the infrared LEDs are on at night and off during the day. The on-time of the infrared LEDs may be limited based on various use cases, as needed, such as when the tractor / trailer is backing up or traveling at speeds below 5-10 mph. As a further example, to improve visibility, both infrared LEDs 38, 40 may be used when backing up and parking, and only one infrared LED may be used when the vehicle is moving forward.

[0039] If the infrared LEDs are on, the controller 42 may be configured to adjust the infrared LED state based on the brightness of the second ROI, for example, by turning off some or all of the infrared LEDs or reducing the power of the infrared LEDs so that they are not turned on at all.

[0040] 4 illustrates a method 60 or algorithm for automatically controlling a vehicle night vision system. An image within a desired field of view is captured (block 62). The desired field of view includes both Class II and Class IV views. At least first and second regions of interest (ROIs) are selected from the captured images (e.g., by cropping from a common image) (block 64). The first ROI indicates the amount of ambient light. It is not necessary to use a single camera to capture the first and second ROIs. It should be understood that one or more cameras may be used. The second ROI is positioned to pick up illumination from light sources such as passing vehicles that may result in oversaturation.

[0041] The luminance of each of the first and second ROIs is determined (block 66). In one example, the image capture unit is configured to capture an RGB image. The controller is configured to convert the RGB images of the first and second ROIs into HSV (hue, saturation, value). Value (V) is used for luminance. An example of a conversion formula is shown below:

[0042] Dividing the R, G, and B values ​​by 255 changes the range from 0..255 to 0..1. R'=R / 255 G'=G / 255 B'=B / 255 C max =max(R', G', B') C min =min(R', G', B') △=C max- C min Hue calculation:

number

number

[0043] A low-pass filter is applied to the HSV values ​​of each of the first and second ROIs to filter out high-frequency noise or fluctuations in V over time. In one example, the low-pass filters for each of the first and second ROIs are different from each other. For example, a second ROI used to detect causes of oversaturation may have a lighter low-pass filter than the V from the first ROI because the V of the second ROI is likely to change faster than the V derived from the first ROI.

[0044] The filtered V of each ROI is ultimately used to determine the brightness of the ROI to adjust the infrared LED state. The infrared LED state is adjusted by changing the infrared LED state only if the HSV brightness exceeds the offset from the desired brightness, providing hysteresis. For example, the infrared LED may be automatically switched on and off based on the V of the first ROI, i.e., the infrared LED may be controlled based on ambient light during the day and at night (block 68). Thus, the infrared LED may be switched off above 0.5 V in the first ROI or switched on below 0.5 V in the second ROI. However, to avoid excessive on / off around 0.5 V, an offset of 0.2 may be used so that once on, the infrared LED is switched off when it reaches 0.3 V. Similarly, once off, the infrared LED may be switched on only when it reaches 0.7 V. Further adjustment of the infrared LED state based on oversaturation detection of the second ROI (block 70) may use a different hysteresis, e.g., an offset of 0.1 V, so that the adjustment of the infrared LED is more active based on the brightness of the second ROI.

[0045] It should be understood that the infrared LEDs need not operate only in an on / off state. That is, the controller may vary the amount of infrared LED illumination other than a fully on infrared LED state or a fully off infrared LED state. For example, in an oversaturation condition detected in the second ROI, the controller 42 may reduce the power of the infrared LEDs, e.g., from 100% power to 30%, 50%, or 70% power.

[0046] The controller 42 may be used to implement the various functions disclosed herein. The controller 42 may include one or more separate units. Furthermore, a portion of the controller 42 may be provided within the vehicle, while another portion of the controller 42 may be located elsewhere. With respect to hardware architecture, such a computing device may include a processor, memory, and one or more input / output (I / O) device interfaces communicatively coupled via a local interface. The local interface may include, for example, but is not limited to, one or more buses and / or other wired or wireless connections. The local interface may also include additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers that enable communication, which are omitted for simplicity. Furthermore, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0047] The controller 42 may be a hardware device for executing software, particularly software stored in a memory, and may be a custom or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller 42, a semiconductor-based microprocessor (in the form of a microchip or chipset), or any device for general-purposely executing software instructions.

[0048] The memory may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard drive, tape, CD-ROM, etc.). Furthermore, the memory may incorporate electronic, magnetic, optical, and / or other types of storage media. The memory may have a distributed architecture where various components are located remotely from each other but are accessible by the processor.

[0049] Software in memory may include one or more separate programs, each containing an ordered list of executable instructions for implementing a logical function. A system component embodied as software may be constructed as a source program, an executable program (object code), a script, or any other entity containing a set of instructions to be executed. If constructed as a source program, the program is translated via a compiler, assembler, interpreter, etc., which may or may not be contained in memory.

[0050] Input / output devices of the present disclosure that may be coupled to the system I / O interface(s) may include, but are not limited to, input devices such as, for example, a keyboard, a mouse, a scanner, a microphone, a camera, a mobile device, a proximity device, etc. Additionally, output devices may include, but are not limited to, a display, a macroclimate device, a microclimate device, etc. Finally, input / output devices may further include devices that communicate as both input and output, such as, but are not limited to, a modulator / demodulator (i.e., for accessing another device, system, or network), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, etc.

[0051] When the controller 42 is in operation, the processor may be configured to execute software stored in the memory, communicate data to and from the memory, and generally control the operation of the computing device in accordance with the software. The software in the memory is read, in whole or in part, by the processor and often buffered within the processor before being executed.

[0052] Also, while particular component arrangements are disclosed in the illustrated embodiments, it should be understood that other arrangements would benefit from the present disclosure. Although a particular sequence of steps is shown, described, and claimed, it should be understood that, unless otherwise indicated, the steps may be performed in any order, separated, or combined, and still benefit from the present invention.

[0053] Although the different examples have specific components shown in the figures, embodiments of the present invention are not limited to those specific combinations. Some components or features of one example can also be used in combination with features or components of another example.

[0054] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. a camera arm having a camera with an image capture unit configured to capture an image of a desired field of view; a display configured to display a desired field of view; an infrared LED (light emitting diode) configured to illuminate at least a portion of a desired field of view; a controller in communication with the image capture unit and the infrared LED; A camera monitor system comprising: the controller is configured to select at least first and second regions of interest (ROIs) from the captured image, the first and second regions of interest corresponding to different first and second portions of the same captured image, the first region of interest indicating an amount of ambient light, the first region of interest being above the horizon and in the sky in the desired field of view, and the second region of interest being rearward along the vehicle having the camera monitoring system; the controller is configured to determine a luminance of each of the first and second regions of interest; the controller is configured to adjust an infrared LED state of the infrared LEDs based on a brightness of the first region of interest; The controller is configured to adjust the infrared LED state based on a brightness of the second region of interest.

2. 2. The system of claim 1, wherein the image capture unit is configured to capture RGB images, and the controller is configured to convert the RGB images of the first and second regions of interest to a Hue, Saturation, Value (HSV) color model.

3. The system of claim 1 , wherein the controller is configured to turn the infrared LEDs from an off state to an on state or from the on state to an off state.

4. The system of claim 1 , wherein the controller is configured to vary the amount of infrared LED illumination other than a fully on infrared LED state or a fully off infrared LED state.

5. the controller is configured to adjust the infrared LED state based on vehicle operating conditions and includes a switch having an automatic position and a manual position; The system of claim 1 , wherein the controller is configured to adjust the infrared LED state in response to the switch being in the automatic position.

6. 1. A method for automatically controlling a vehicle night vision system, comprising: capturing an image of a desired field of view with at least one image capture unit; selecting, by a controller, at least first and second regions of interest (ROIs) from the captured image, the first and second regions of interest corresponding to different first and second portions of the same captured image, the first region of interest being above the horizon and in the sky in the desired field of view, and the second region of interest being rearward along the vehicle having the image capture unit; determining, by the controller, a brightness of each of the first and second regions of interest, the first region of interest indicating an amount of ambient light; adjusting, by the controller, an infrared LED state of an infrared LED based on a brightness of the first region of interest; adjusting the infrared LED state based on the brightness of the second region of interest; A method comprising:

7. The method of claim 6 , wherein the desired field of view includes both Class II and Class IV views.

8. 7. The method of claim 6, wherein the step of capturing an image includes capturing an RGB image, and the step of determining the luminance includes converting the RGB images of the first and second regions of interest to HSV (hue, saturation, value).

9. The method of claim 8 , wherein the step of determining luminance comprises calculating a median brightness value.

10. The method of claim 9 , wherein determining the luminance comprises applying a low pass filter to the brightness for each of the first and second regions of interest.

11. The method of claim 10 , wherein the low pass filters for each of the first and second regions of interest are different from each other.

12. 11. The method of claim 10, wherein the determining the brightness step comprises comparing the HSV brightness to a desired brightness, and the adjusting the infrared LED state step comprises changing the infrared LED state only if the brightness exceeds an offset from the desired brightness.

13. The method of claim 6 , wherein adjusting the infrared LED state includes automatically turning the infrared LED on or off.

14. 14. The method of claim 13, wherein adjusting the infrared LED state comprises turning the infrared LED from an off state to an on state or from an on state to an off state.

15. 14. The method of claim 13, wherein adjusting the infrared LED state comprises varying an amount of infrared LED illumination other than a fully on or fully off state of the infrared LED.

16. The method of claim 6 , wherein adjusting the infrared LED state is performed based on a vehicle operating state.

17. a switch having an automatic position and a manual position; 7. The method of claim 6, wherein the step of adjusting the infrared LED state is performed in response to the switch being in the automatic position.

Images