Automatic infrared LED control for camera monitor system

Abstract

To provide a camera monitor system and a method for improving driver visibility in a low light situation in a camera monitor system (CMS) using an infrared LED for night vision.SOLUTION: A camera monitor system CMS 30 includes a camera arm 16 that has a camera 20 with an image capture unit 36 that is configured to capture an image of a desired field of view. A display 18 is configured to display the desired field of view. Infrared (IR) LEDs 38 and 40 illuminate IR LED illumination areas 26 and 28 that are at least portions of the desired field of view. A controller is capable of communicating with the image capture unit and the IR LEDs, and selects at least a first region of interest (ROI) and a second ROI from the captured image. The first ROI is indicative of an amount of ambient light. The controller adjusts the IR LED state based on first ROI luminance and / or second ROI luminance.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a CMS system using infrared LEDs for night vision.

Background Art

[0002] Using infrared LEDs can improve the visibility of drivers in low light conditions. The infrared LEDs are controlled so as not to be continuously on to manage heat. When using a camera monitoring system with a night vision function, the image displayed to the driver inside the vehicle may become overly saturated in color, and objects or people may not be clearly visible.

Summary of the Invention

[0003] In one exemplary embodiment, a camera monitoring system includes a camera arm having a camera with an image capture unit configured to capture an image of a desired field of view. The system further includes a display configured to display the desired field of view, an infrared LED configured to illuminate at least a portion of the desired field of view, and a controller in communication with the image capture unit and the infrared LED. The controller is configured to select at least first and second regions of interest (ROIs) from the captured image. The first ROI indicates the amount of ambient light. The controller is configured to determine the luminance of each of the first and second ROIs. The controller is configured to adjust the infrared LED state of the infrared LED based on the luminance of the first ROI, and the controller is configured to adjust the infrared LED state based on the luminance of the second ROI.

[0004] In any further embodiment of the above, the first ROI and the second ROI correspond to different first and second portions of the same image, the first ROI is above the horizontal line and in the sky in the desired field of view, and the second ROI is behind along the vehicle having the camera monitoring system.

[0005] In any further embodiment described above, the image capture unit is configured to capture RGB images, and the controller is configured to convert the RGB images of the first and second ROIs to HSV.

[0006] In any further embodiment described above, the controller is configured to turn the infrared LED from an off state to an on state, or from an on state to an off state.

[0007] In any further embodiment described above, the controller is configured to vary the amount of infrared LED illumination other than a fully on infrared LED state or a fully off infrared LED state.

[0008] In any further embodiment described above, the controller is configured to adjust the infrared LED state based on the vehicle operating state and includes a switch having an automatic position and a manual position. The controller is configured to adjust the infrared LED state in response to the switch being in the automatic position.

[0009] In another exemplary embodiment, a method for automatically controlling a vehicle night vision system includes the steps of: capturing an image in a desired field of view; selecting at least first and second regions of interest (ROIs) from the captured image, wherein the first ROI indicates the amount of ambient light; determining the brightness of the first and second ROIs, wherein the first ROI indicates the amount of ambient light; adjusting the infrared LED state of an infrared LED based on the brightness of the first ROI; and adjusting the infrared LED state based on the brightness of the second ROI.

[0010] In any further embodiment described above, the first and second ROIs correspond to different first and second portions of the same captured image.

[0011] In any further embodiment described above, the first ROI is above the horizon and in the sky in the desired field of view, and the second ROI is behind the vehicle having the night vision system.

[0012] In any further embodiment described above, the desired field of view includes both Class II and Class IV views.

[0013] In any further embodiment described above, the step of capturing the image includes capturing an RGB image, and the step of determining the brightness includes converting the RGB images of the first and second ROIs to HSV (hue, saturation, lightness).

[0014] In any further embodiment described above, the step of determining the luminance includes calculating the median value of the brightness.

[0015] In any further embodiment described above, the step of determining the brightness includes applying a low-pass filter to the brightness for each of the first and second ROIs.

[0016] In any further embodiment described above, the low-pass filters for the first and second ROIs are different from each other.

[0017] In any further embodiment described above, the step of determining the brightness includes comparing the brightness of the HSV with a desired brightness, and the step of adjusting the infrared LED state includes changing the infrared LED state only if the brightness exceeds an offset from the desired brightness.

[0018] In any further embodiment described above, the step of adjusting the infrared LED state includes automatically turning the infrared LED on or off.

[0019] In any of the further embodiments described above, the step of adjusting the infrared LED state includes turning the infrared LED from an off state to an on state or from an on state to an off state.

[0020] In any of the further embodiments described above, the step of adjusting the infrared LED state includes changing the amount of infrared LED illumination other than a full on state or a full off state of the infrared LED.

[0021] In any of the further embodiments described above, the step of adjusting the infrared LED state is performed based on the operating state of the vehicle.

[0022] In any of the further embodiments described above, the method includes a switch having automatic and manual positions, and the step of adjusting the infrared LED state is performed in response to the switch being in the automatic position.

[0023] The present disclosure can be further understood by reference to the following detailed description when considered in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0024] [Figure 1A] It is a schematic front view of a commercial truck equipped with a camera monitoring system (CMS) used to provide at least Class II and Class IV views. [Figure 1B] It is a schematic top view of a commercial truck equipped with a camera monitoring system that provides views of Class II, Class IV, Class V, and Class VI. [Figure 2] It is a schematic diagram of a CMS having night vision capabilities according to the system and method of the present disclosure. [Figure 3] Figures 3A and 3B respectively show views displayed during daytime and nighttime using an image capture unit having first and second regions of interest. [Figure 4]An example of a method for controlling an infrared LED using information from first and second regions of interest.

Best Mode for Carrying Out the Invention

[0025] The embodiments, examples and alternatives, claims, or the following description and drawings of the above paragraphs may be adopted independently or in any combination, including any of their various aspects or their respective individual features. Features described in relation to one embodiment are applicable to all embodiments unless such features are incompatible. Similar reference numbers and symbols in the various drawings indicate similar elements.

[0026] Schematic views of a commercial vehicle 10 are shown in FIGS. 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or the trailer 14 may have any configuration. Although a commercial truck is contemplated in the present disclosure, the systems of the present disclosure may be applied to other types of vehicles. The vehicle 10 incorporates a camera monitoring system (CMS) 30 (FIG. 2), to which camera arms 16a, 16b (generally "16") on the driver's and passenger sides are attached outside the vehicle cab 12. If desired, the camera arms 16a, 16b may include conventional mirrors integrated therewith, or the CMS 30 may be used to completely replace the side view mirrors. In additional examples, multiple camera arms 16 may be included on each side, and each arm 16 may house one or more cameras and / or mirrors.

[0027] Each camera arm 16a, 16b includes a base 32 fixed to the cab 12, for example, as shown in Figure 2. A swivel arm 34 is supported by the base 34 and may be articulated relative to the base, either manually or using an electric retraction mechanism. Returning to Figure 1B, at least one rear-facing camera 20a, 20b (generally "20") is positioned within each camera arm, respectively. The external cameras 20a, 20b each provide external field of view FOVEX1, FOVEX2, each including at least one of Class II and Class IV views (Figure 1B), which are legally defined views in the commercial truck industry. A Class II view on a given side of the vehicle 10 is a subset of the Class IV views on the same side of the vehicle 10. In one example, both Class II and Class IV views are provided by a single camera that provides a wide-angle view. If necessary, multiple cameras may be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in the European R46 Act, and the United States and other countries also have similar driver visibility requirements for commercial trucks. The reference to “class” views is not intended to be limiting, but rather to be an example of the types of views that can be provided to a display by a particular camera. Each arm 16a, 16b may also provide a housing that encloses electronics configured to provide various features of the CMS 30.

[0028] First and second video displays 18a, 18b (generally "18") are positioned on or near the A-pillars in the driver's seat 12, on the driver's side and passenger's side, respectively, to display Class II and Class IV views on each side of the vehicle 10, thereby providing a rear view along the vehicle 10 captured by external cameras 20a, 20b. The Class II and Class IV views may be provided by cropping a portion of the image from a wide-angle camera.

[0029] If images of Class V and Class VI views are also required, a camera housing 16c and a camera 20c may be positioned in front of or near the front of the vehicle 10 to provide these views (Figure 1B). A third display 18c, positioned in the driver's seat 12 near the upper center of the windshield, can be used to display Class V and Class VI views forward of the vehicle 10 to the driver. Displays 18a, 18b, and 18c (generally, displays 18) are oriented towards the driver area 24 in the driver's seat 22 where the driver is seated in the driver's seat 26. The position, size, and (multiple) fields of view streamed to a particular display may differ from the configurations described herein, but still constitute the invention of this disclosure.

[0030] If a view of Class VIII is required, camera housings can be positioned on the sides and rear of the vehicle 10 to provide a field of view that includes part or all of the Class VIII zone of the vehicle 10. In such an example, the third display 18c may include one or more frames displaying the Class VIII view. Alternatively, additional displays can be added near the first, second, and third displays 18a, 18b, and 18c to provide dedicated displays for providing the Class VIII view.

[0031] It should be understood that more or fewer displays may be used than described in general terms, that images from multiple cameras may be combined on a single display, or that images from a particular field of view may be provided on separate displays separated from other images.

[0032] The area behind a trailer is a blind spot common to all vehicles, but it is a particularly significant blind spot for commercial trucks. Therefore, as shown in Figure 1B, it is desirable to use sensors such as camera 20d to give the driver some awareness of unseen objects at the rear of the trailer. A challenge in using cameras at the rear of a trailer is the long wiring required to transmit the video signal to the display in the driver's cab. Dedicated wiring would significantly increase the system cost. Furthermore, the image needs to be transmitted with minimal or no delay so that the object is displayed in real time.

[0033] Figure 2 shows a night vision system 30. The system 30 includes an image capture unit 36 ​​configured to capture a desired field of view of a camera 20. The camera 20 is positioned on a pivot 34 of a camera arm 16, which is articulated to a fixed base portion 32 fixed to the vehicle's driver's cab 12. The camera arm 16 also has one or more infrared LEDs 38, 40. The infrared LEDs may be single infrared LEDs or arrays of multiple infrared LEDs, and they may be controlled collectively or independently of each other. In one example, a first infrared LED array 38 may provide one night vision illumination area 28, and a second array 40 may provide another infrared LED illumination area 26, which may illuminate the side of the vehicle 10 as shown in Figure 1B. It should be understood that a single infrared LED or an array of infrared LEDs may be used instead of the two arrays shown in the figure.

[0034] Returning to Figure 2, the controller 42, which may be located in the vehicle's driver's cab 12, communicates with the image capture unit 36 ​​and the infrared LEDs 38 and 40. The controller 42 may include video processing to display the images captured from the image capture unit 36 ​​in a desired format on a display 18, such as a Class II or Class IV view.

[0035] The controller 42 communicates with various input / output devices. The controller 42 may receive information about the vehicle's operating status from vehicle components via the CAN bus. For example, the gear position switch 48 may indicate whether the vehicle is in forward gear or reverse gear. The vehicle speed sensor 50 provides vehicle speed information. An example of another input device is a switch that can be moved between Off, Automatic, and Manual. In the Off position, the infrared LED switches to the Off infrared LED state and becomes inoperable. In the Manual position, driving can switch the infrared LED to the On infrared LED state, regardless of whether the infrared LED is turned on in Automatic operating mode. In the Automatic position, the controller 42 uses algorithm 60, which is described in more detail below and summarized in the manner shown in Figure 4.

[0036] Referring to Figures 3A and 3B, the display 18 includes at least two regions of interest (ROIs) which are shown as cropped portions of the same image provided by the image capture unit 36. These regions are not displayed on the display, Figures 3A and 3B show the possible locations where regions may be positioned to provide the described functionality. More than two ROIs may be used as needed. Controller 42 is configured to determine the average luminance of the first and second ROIs 56, 58 described above. The luminance values ​​may be normalized between 0 and 1, where 0 is pitch black and 1 is full light. The intermediate value of 0.5 may be used, for example, as a dividing point between daytime and nighttime. The value of 0.5 may also be used to determine whether an oversaturation state can occur at night, for example, whether the luminance in the second ROI 58 is less than 0.5 at night. Other values ​​may be used as needed.

[0037] The first and second ROIs 56 and 58 are located above the horizon and in the sky, along the vehicle, for example, along the trailer 14, and provide a portion of the image to the rear. The first ROI 56 is intended to provide an indication of the amount of ambient light, i.e., daytime or nighttime. In the illustrated example, the average brightness of the portion of the captured image within the first ROI 56 is 0.77, indicating daytime (Figure 3A), and a value of 0.23 indicates nighttime (Figure 3B). The second ROI 58 is intended to capture illumination that can saturate any image or object in the field of view when the infrared LED is turned on, for example, when a vehicle with headlights is overtaking the tractor / trailer.

[0038] When switch 46 is in the automatic position, controller 42 is configured to adjust the infrared LED state of infrared LEDs 38 and / or 40 based on the brightness of the first ROI. That is, the infrared LEDs are turned on at night and turned off during the day. The on time of the infrared LEDs may be limited as needed, based on various use cases, for example, when the tractor / trailer is backing up or traveling at a speed of less than 5-10 mph. As a further example, both infrared LEDs 38 and 40 may be used when backing up and parking to improve visibility, and only one of the infrared LEDs may be used when the vehicle is moving forward.

[0039] When the infrared LEDs are turned on, the controller 42 may be configured to adjust the state of the infrared LEDs based on the brightness of the second ROI. For example, some or all of the infrared LEDs may be turned off, or the power of the infrared LEDs may be reduced so that they are not turned on at all.

[0040] Figure 4 shows a method 60 or algorithm for automatically controlling a vehicle night vision system. Images within a desired field of view are captured (block 62). The desired field of view includes both Class II and Class IV views. At least first and second regions of interest (ROIs) are selected from the captured images (e.g., by cropping from a common image) (block 64). The first ROI indicates the amount of ambient light. It is not necessary to capture the first and second ROIs using a single camera. It should be understood that more than one camera may be used. The second ROI is positioned to capture lighting from light sources such as passing vehicles, which may result in oversaturation.

[0041] The luminance values ​​for the first and second ROIs are determined (block 66). In one example, the image capture unit is configured to capture an RGB image. The controller is configured to convert the RGB images of the first and second ROIs to HSV (hue, saturation, lightness). Lightness (V) is used relative to luminance. An example of the conversion formula is shown below.

[0042] Dividing the R, G, and B values ​​by 255 changes the range from 0.255 to 0.1. R' = R / 255 G' = G / 255 B' = B / 255 C max =max(R', G', B') C min =min(R', G', B') △=C max- C min Hue calculation:

number

number

[0043] Low-pass filters are applied to the brightness of the HSV of each of the first and second ROIs to filter out high-frequency noise or fluctuations in V over time. In one example, the low-pass filters for the first and second ROIs are different from each other. For example, the second ROI used to detect the cause of oversaturation may have a lighter low-pass filter than the V from the first ROI, because the V of the second ROI is likely to change faster than the V derived from the first ROI.

[0044] The filtered V of each ROI is used to determine the brightness of the ROI for adjusting the infrared LED state. The infrared LED state is adjusted by changing the infrared LED state only when the brightness of the HSV provides hysteresis beyond the offset from the desired brightness. For example, the infrared LED may be automatically switched on and off based on the V of the first ROI, i.e., the infrared LED may be controlled based on daytime and nighttime ambient light (block 68). Thus, the infrared LED may be switched off above 0.5V in the first ROI or switched on below 0.5V in the second ROI. However, to avoid excessive on / off around 0.5V, an offset of 0.2 may be used so that once the infrared LED is on, it switches off when it reaches 0.3V. Similarly, once the infrared LED is off, it may only be switched on when it reaches 0.7V. In the further adjustment of the infrared LED state based on the detection of oversaturation of the second ROI (block 70), a different hysteresis, e.g., a 0.1V offset, may be used so that the adjustment of the infrared LED is more active based on the brightness of the second ROI.

[0045] It should be understood that infrared LEDs do not have to operate only in on / off states. That is, the controller may vary the amount of infrared LED illumination in states other than fully on or fully off. For example, in a state of oversaturation detected in the second ROI, the controller 42 may reduce the power of the infrared LEDs from, for example, 100% power to 30%, 50%, or 70% power.

[0046] The controller 42 may be used to implement various functions disclosed herein. The controller 42 may include one or more separate units. Furthermore, part of the controller 42 may be located in a vehicle, and other parts of the controller 42 may be located elsewhere. With respect to the hardware architecture, such a computer device may include a processor, memory, and one or more input / output (I / O) device interfaces, which are communicatively coupled via a local interface. The local interface may include, but is not limited to, one or more buses and / or other wired or wireless connections. The local interface may include additional elements, such as controllers, buffers (caches), drivers, repeaters, and receivers that enable communication, although these are omitted for simplicity. Furthermore, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components.

[0047] The controller 42 may be a hardware device for executing software, particularly software stored in memory. The controller 42 may be a custom-made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the controller 42, a semiconductor-based microprocessor (in the form of a microchip or chipset), or any device for executing software instructions in a general-purpose manner.

[0048] Memory may include any one or combination of volatile memory elements (e.g., random access memory (RAM such as DRAM, SRAM, SDRAM, VRAM, etc.)) and / or non-volatile memory elements (e.g., ROM, hard drives, tapes, CD-ROMs, etc.). Furthermore, memory may incorporate electronic, magnetic, optical, and / or other types of storage media. Memory may have a distributed architecture in which various components are located remotely from one another but can be accessed from the processor.

[0049] Software in memory may contain one or more separate programs, each containing an ordered list of executable instructions for implementing logical functions. System components embodied as software may be constructed as source programs, executable programs (object code), scripts, or any other entity containing a set of instructions to be executed. If constructed as a source program, the program is translated through compilers, assemblers, interpreters, etc., which may or may not be contained in memory.

[0050] The input / output devices of this disclosure, which can be coupled to (multiple) system I / O interfaces, may include, but are not limited to, input devices such as keyboards, mice, scanners, microphones, cameras, mobile devices, and proximity devices. Furthermore, output devices may include, but are not limited to, displays, macroclimate devices, microclimate devices, and so on. Finally, the input / output devices may further include, but are not limited to, devices that communicate as both inputs and outputs, such as modulators / demodulators (modems, i.e., for accessing another device, system, or network), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, and so on.

[0051] When the controller 42 is operating, the processor may be configured to execute software stored in memory, communicate data with memory, and generally control the operation of the computer device according to the software. The software in memory is read in whole or in part by the processor, often buffered within the processor, and then executed.

[0052] Furthermore, while specific configurations of components are disclosed in the exemplary embodiments, it should be understood that other configurations may also benefit from this disclosure. Although specific step sequences are shown, described, and described in the claims, it should be understood that, unless otherwise specified, the steps may be performed in any order, separated or combined, and still benefit from the present invention.

[0053] Different examples may have specific components shown in the figures, but embodiments of the present invention are not limited to any particular combination thereof. Some components or functions of one example may be used in combination with functions or components of another example.

[0054] While exemplary embodiments have been disclosed, those skilled in the art will recognize that certain modifications fall within the scope of the claims. Therefore, the following claims should be considered in order to determine their true scope and content.

Claims

1. a camera arm having a camera with an image capture unit configured to capture an image of a desired field of view; a display configured to display a desired field of view; an infrared LED (light emitting diode) configured to illuminate at least a portion of a desired field of view; a controller in communication with the image capture unit and the infrared LED; A camera monitor system comprising: the controller is configured to select at least first and second regions of interest (ROIs) from the captured image, the first and second regions of interest corresponding to different first and second portions of the same captured image, the first region of interest indicating an amount of ambient light, the first region of interest being above the horizon and in the sky in the desired field of view, and the second region of interest being rearward along the vehicle having the camera monitoring system; the controller is configured to determine a luminance of each of the first and second regions of interest; the controller is configured to adjust an infrared LED state of the infrared LEDs based on a brightness of the first region of interest; The controller is configured to adjust the infrared LED state based on a brightness of the second region of interest.

2. 2. The system of claim 1, wherein the image capture unit is configured to capture RGB images, and the controller is configured to convert the RGB images of the first and second regions of interest to a Hue, Saturation, Value (HSV) color model.

3. The system of claim 1 , wherein the controller is configured to turn the infrared LEDs from an off state to an on state or from the on state to an off state.

4. The system of claim 1 , wherein the controller is configured to vary the amount of infrared LED illumination other than a fully on infrared LED state or a fully off infrared LED state.

5. the controller is configured to adjust the infrared LED state based on vehicle operating conditions and includes a switch having an automatic position and a manual position; The system of claim 1 , wherein the controller is configured to adjust the infrared LED state in response to the switch being in the automatic position.

6. 1. A method for automatically controlling a vehicle night vision system, comprising: capturing an image of a desired field of view with at least one image capture unit; selecting, by a controller, at least first and second regions of interest (ROIs) from the captured image, the first and second regions of interest corresponding to different first and second portions of the same captured image, the first region of interest being above the horizon and in the sky in the desired field of view, and the second region of interest being rearward along the vehicle having the image capture unit; determining, by the controller, a brightness of each of the first and second regions of interest, the first region of interest indicating an amount of ambient light; adjusting, by the controller, an infrared LED state of an infrared LED based on a brightness of the first region of interest; adjusting the infrared LED state based on the brightness of the second region of interest; A method comprising:

7. The method of claim 6 , wherein the desired field of view includes both Class II and Class IV views.

8. 7. The method of claim 6, wherein the step of capturing an image includes capturing an RGB image, and the step of determining the luminance includes converting the RGB images of the first and second regions of interest to HSV (hue, saturation, value).

9. The method of claim 8 , wherein the step of determining luminance comprises calculating a median brightness value.

10. The method of claim 9 , wherein determining the luminance comprises applying a low pass filter to the brightness for each of the first and second regions of interest.

11. The method of claim 10 , wherein the low pass filters for each of the first and second regions of interest are different from each other.

12. 11. The method of claim 10, wherein the determining the brightness step comprises comparing the HSV brightness to a desired brightness, and the adjusting the infrared LED state step comprises changing the infrared LED state only if the brightness exceeds an offset from the desired brightness.

13. The method of claim 6 , wherein adjusting the infrared LED state includes automatically turning the infrared LED on or off.

14. 14. The method of claim 13, wherein adjusting the infrared LED state comprises turning the infrared LED from an off state to an on state or from an on state to an off state.

15. 14. The method of claim 13, wherein adjusting the infrared LED state comprises varying an amount of infrared LED illumination other than a fully on or fully off state of the infrared LED.

16. The method of claim 6 , wherein adjusting the infrared LED state is performed based on a vehicle operating state.

17. a switch having an automatic position and a manual position; 7. The method of claim 6, wherein the step of adjusting the infrared LED state is performed in response to the switch being in the automatic position.

Images