Image processing system for camera monitor system

Abstract

A method for processing a digital image includes receiving an image from a camera, inverting color data of the image, and bit-shifting each pixel in the image such that the total bits per color (BPC) of each pixel is reduced while preserving the data elements of the image. The bit-shifted image is provided to a computer vision system.

Description

[Technical Field]

[0001] The present disclosure relates to a camera monitor system (CMS) for a vehicle, and more particularly to a method for processing digital images to reduce the computational load in the CMS. [Background technology]

[0002] Camera monitor systems (CMS), such as mirror replacement systems and camera systems that supplement mirror views, are being utilized in vehicles to enhance the vehicle operator's ability to view the surrounding environment. CMS utilize one or more cameras to provide the vehicle operator with an enhanced field of view. In some instances, the camera system covers a wider field of view than a traditional mirror or includes views not fully available through a traditional mirror. In other instances, a CMS provides image analytics that can be used for driver assistance systems and automated or semi-automated vehicle operation.

[0003] Images provided via cameras within the CMS can be utilized by a processor within the CMS to detect elements of the environment and of the vehicle in an image processing-based detection process, which is computationally intensive and time consuming. Summary of the Invention

[0004] An exemplary method for processing a digital image includes receiving an image from a camera, inverting color data of the image, bit-shifting each pixel in the image to reduce the total bits per color (BPC) of each pixel while preserving the data elements of the image, and providing the shifted image to a computer vision system.

[0005] In another example of the above method for processing a digital image, the step of bit-shifting each pixel includes reducing the total BPC of each pixel by a predetermined number of pixels.

[0006] In another example of the above method for processing a digital image, the step of bit-shifting each pixel includes reducing the total BPC of each pixel from 8 BPC to 4 BPC.

[0007] Another example of the above method for processing digital images further comprises repeating the method of claim 1 for a set of consecutive images in a video feed.

[0008] In another example of the above method for processing a digital image, bit-shifting each pixel in the image to reduce the total BPC for each pixel includes adjusting the number of BPCs to reduce at each pixel based on a feedback loop for each successive image in the set of successive images.

[0009] In another example of the above method for processing a digital image, the step of inverting the color data includes performing a nonlinear logarithmic transformation on the pixel bit-depth data using the formula log n(x), where N is a logarithm base value in the range of 2 to 10.

[0010] In another example of the above method for processing a digital image, bit-shifting each pixel in the image includes shifting image detail bits from most significant image bits to least significant image bits.

[0011] In another example of the above method for processing a digital image, the shifted image includes sufficient data elements to perform at least one process of the computer vision system.

[0012] In another example of the above method for processing a digital image, the data elements include contrast lines and edge lines.

[0013] In another example of the above method for processing a digital image, the shifted image comprises reduced color and / or pattern data.

[0014] In one exemplary embodiment, a camera monitor system (CMS) for a vehicle includes at least one camera providing a video feed to a CMS controller, the CMS controller including a memory that stores instructions that cause the CMS controller to perform a video feed pre-processing method configured to: bit-shift each pixel in the video feed to reduce a total bits per color (BPC) for each pixel while preserving data elements of the image; and providing the reduced BPC video feed to at least one computer vision system.

[0015] In another example of the above CMS for a vehicle, the CMS controller is further configured to invert color data of the video feed before performing the bit shifting.

[0016] In another example of any of the above-described CMS for a vehicle, inverting the color data includes performing a non-linear logarithmic transform on pixel bit-depth data.

[0017] In another example of any of the above-described vehicle CMS, the video feed is an RGB888 video feed.

[0018] In another example of any of the above-described vehicle CMSs, bit-shifting each pixel in the video feed includes shifting the video feed from the RGB888 video feed to an RGB444 video feed.

[0019] In another example of any of the above-described vehicle CMSs, bit-shifting each pixel in the video feed includes reducing the video feed from the RGB888 to an RGB feed having a pixel size of less than 8 bits, where the resulting pixel size is variable.

[0020] In another example of any of the above-described vehicle CMS, the resulting pixel size is controlled via a feedback loop that includes the bit shifting process and a feedback output from the computer vision system.

[0021] These and other features of the present invention can be best understood from the following specification and drawings. [Brief explanation of the drawings]

[0022] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0023] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera monitor system (CMS) used to provide at least Class II and Class IV views.

[0024] [Figure 1B] FIG. 1 is a schematic top view of a commercial truck equipped with a camera monitor system providing Class II, Class IV, Class V, and Class VI views.

[0025] [Figure 2] FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an interior camera.

[0026] [Figure 3] 1 illustrates an image processing method configured to pre-process images for use in a computer vision system.

[0027] [Figure 4] 4 illustrates a bit-shifting process for reducing the Bits Per Color (BPC) size of an image to a predetermined amount within the method of FIG. 3.

[0028] [Figure 5] 4 illustrates a bit-shifting process for reducing the image's bits-per-color (BPC) size by a variable amount within the method of FIG. 3.

[0029] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE INVENTION

[0030] Schematic diagrams of a commercial vehicle 10 are shown in FIGS. 1A and 1B. FIG. 2 is a schematic top perspective view of the cab of the vehicle 10, including a display and an interior camera. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. It should be understood that the vehicle cab 12 and / or trailer 14 may be of any configuration. While commercial trucks are contemplated in this disclosure, the present invention is applicable to other types of vehicles. The vehicle 10 incorporates a camera monitor system (CMS) 15 (FIG. 2) that includes driver and passenger side camera arms 16a, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, each arm housing one or more cameras and / or mirrors.

[0031] Each camera arm 16a, 16b includes a base fixed to, for example, the cab 12. A pivoting arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20a, 20b is disposed within each camera arm. Each exterior camera 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1b), which are legally defined views in the commercial trucking industry. EX1 , FOV EX216a, 16b. If desired, multiple cameras may be used in each camera arm 16a, 16b to provide these views. For example, Class II and Class IV views are defined in the European R46 legislation, and the United States and other countries have similar driving visibility requirements for commercial trucks. References to "class" views are not intended to be limiting, but rather as an illustration of the type of view provided to the display by a particular camera. Each arm 16a, 16b may also provide a housing enclosing electronics configured to provide various features of the CMS 15.

[0032] In one example, in addition to cameras 20a, 20b in camera arms 16a, 16b, CMS includes at least a rear-facing camera 60 and an interior trailer camera 62. Rear-facing camera 60 captures a view of the exterior of trailer 14, while interior camera 62 captures a view of the interior of trailer 14, including objects loaded on trailer 14. In an alternative example, either camera 60, 62 may be a camera incorporated into a secondary system connected to CMS 15 and configured to provide a video feed to controller 15, and the following description may function similarly.

[0033] First and second video displays 18a, 18b are positioned on the driver's side and passenger's side, respectively, within the vehicle cab 12 on or near the A-pillars 19a, 19b and display Class II and Class IV views on each side of the vehicle 10, which provide rear-facing views along the vehicle 10 captured by exterior cameras 20a, 20b.

[0034] If video of Class V and / or Class VI views is also required, a camera housing 16c and camera 20c may be positioned at or near the front of the vehicle 10 to provide these views (FIG. 1b). A third display 18c located within the cab 12 near the top center of the windshield can be used to display Class V and Class VI views forward of the vehicle 10 to the driver. Displays 18a, 18b, 18c face a driver area 24 within the cab 22, where the driver is seated in a driver's seat 26. The position, size, and field of view(s) streamed to a particular display may vary from the configurations described herein and still encompass the invention of this disclosure.

[0035] If video of a Class VIII view is desired, camera housings can be positioned on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of the vehicle 10. In such an example, the third display 18c can include one or more frames that display the Class VIII view. Alternatively, additional displays can be added near the first, second, and third displays 18a, 18b, 18c to provide a dedicated display that provides the Class VIII view.

[0036] CMS 15 uses images generated by cameras 20a, 20b, and 20c for both mirror replacement / supplemental views and object detection and driver assistance functions. While mirror replacement / supplemental views require color and pattern data for the vehicle operator, it is understood that certain image elements and characteristics (e.g., color variations and patterns) that humans rely on to distinguish objects for edge detection, automated driver assistance systems, object detection, and other digital image analysis are not useful for computer analysis. This is because machines, such as computer processors, do not "see" images the way humans do. Rather, machines see contrasts between objects and features and use those contrasts to analyze images.

[0037] In existing systems, image enhancements and modifications are typically constructed with human visual perception in mind. As a result, enhancement techniques are designed to preserve and enhance colors and patterns that are useful to human analysis. These colors and patterns typically offer little benefit to computer analysis, rendering techniques that preserve or enhance them useless. Another way of expressing this concept is that enhancement techniques used in existing systems focus on aesthetics and detail, while computer analysis systems focus on data.

[0038] In contrast to existing enhancement techniques, the CMS 15 described herein uses an image processing method that reduces the bit size of each pixel in an image by reducing the pixel quantization level. Reducing the bit size of each pixel allows for faster processing of video frames with lower computational load using edge contrast transfer functions and similar computer analyses. Reduced pixel quantization is achieved by shifting detail bits from the high end (most significant bits) of the image spectrum to the low end (least significant bits) by compressing the color data. This shift reduces the amount of data contained within each pixel without removing or altering image detail, such as contrast, required from CMS-based digital analysis.

[0039] With continued reference to Figures 1-2, Figure 3 schematically illustrates a high-level process flow of a method 200 for converting an RGB888 (red, green, blue image with 8 bits per color) image input 210 from a video source to CMS 15 into an RGB444 (red, green, blue image with 4 bits per color) image 212 for analysis by one or more controllers and processes within CMS 15.

[0040] First, the RGB888 image 210 is processed using an image enhancement unit 220, which tone maps the image. Tone mapping is a technique used in image processing and computer graphics to map one color or set of colors to another. Tone mapping is typically used on images intended for human vision to approximate the appearance of high-dynamic color range images in media with a more limited dynamic range. Tone mapping algorithms compress the upper and lower bands of data for each pixel into a mid-band while maintaining clear separation between objects. This compression effectively condenses the colors in the image without substantially changing the contrast data. Tone mapping results in less color variation and potentially a loss of pattern detail in the image.

[0041] Tone mapping algorithms first perform a nonlinear logarithmic transformation on the pixel bit-depth data. In tone mapping algorithms, the image signal is compressed using the log n(x) formula, where N is a logarithmic base value, typically between 2 and 10. Each step increases the compression. X is the image signal value for each pixel. The nonlinear logarithmic transformation compresses the high bit-depth values ​​(e.g., upper and lower bands) while preserving the lower bit values ​​and maintaining the mid-spectrum of bits spread according to a linear model.

[0042] After applying tone mapping, the image is normalized using a normalization function, which involves converting the log-transformed image values ​​back to full-bit image values.

[0043] The output of the normalization function 230 is provided to an inverter 240. The inverter 240 inverts the color values ​​of the image by inverting the data according to M=(N-255), N>=0, where M is the inverted output and N is the logarithmic input (i.e., the normalized value of the output of the enhancement process 220). Because extrinsic information usually occupies the most significant bits in typical image processing formats, the inversion function enhances edge information in the image and discards extrinsic color information from the image.

[0044] After inverting the image data, high-frequency data is removed from the image using a bit-shifting process 250. The bit-shifting process 250 reduces the number of quantization levels for each pixel of an RGB image from 8 bpc to as few bits as possible while preserving the image's data elements (e.g., contrast and edge information). Figure 4 shows a first example 350 of this process, which reduces the bit size of an image by a predefined amount. The process 350 begins by shifting the bpc to 7 bpc in shift 352. The shifting is repeated to reduce from 7 to 6 bpc (shift 354), 6 to 5 bpc (shift 356), and 5 to 4 bpc (shift 358). The output 359 is a 4 bpc image value, which is then normalized again in normalization process 260 (shown in Figure 3).

[0045] Each bit shift 352, 354, 356, 358 reduces the brightness level of the image while retaining all necessary information data from the image that can be used to detect objects and people in the environment. The three-channel input (RGB) retains significant enough redundant data to fully operate all computer-based vision processing systems. Human-centric information such as color gradients and patterns is reduced or eliminated, but this reduction does not affect the functionality of computer-based vision systems.

[0046] In some examples, such as the bit shifting process 450 shown in FIG. 5, a feedback loop may be implemented to allow the amount of bit shifting 250 to be variable rather than predefined by four shifts as shown in FIG. 4. In the variable bit shifting process 450, the process first performs a continuous bit shifting 452 operation for a predefined BPC reduction, as shown in the example of FIG. 4. The output of the bit shifting process 450 is provided to a machine vision process 460 that uses the image. The machine vision process 460 analyzes the image using any conventional computer-based image analysis technique and provides a conventional output 462 corresponding to the analysis technique.

[0047] In addition to the conventional output 462, the machine vision system 460 provides a feedback output 464 to the variable bit shifting process 450. In one example, the feedback output 464 can be one of three states: high, low, or good. A high state indicates that the resolution provided to the machine vision process 460 is higher than necessary, and each successive bit shifting operation of the bit shifting process 460 may result in additional bit shifting. A low state indicates that the resolution provided is so low that excessive bit shifting would result in data elements being lost, and indicates that the variable bit shifting process should reduce the amount of bit shifting. A good state indicates that the amount of bit shifting should not be increased or decreased. The amount of shifting performed by the bit shifting process 450 is adjusted according to the state of the feedback signal 464.

[0048] In a further variation of the feedback loop 464 shown in FIG. 5, the high or low state is accompanied by a magnitude that indicates how much higher or lower the resolution is compared to the good level, and the magnitude of the feedback loop affects the number of bit shifts that are added or removed from the next iteration of the bit shifting process 460.

[0049] While exemplary embodiments have been disclosed, those of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. 1. A method for processing a digital image, comprising: a) receiving an image from a camera; b) inverting the color data of the image; c) bit-shifting each pixel in the image of inverted color data to reduce the total BPC (bits per color) of each pixel while preserving the data elements of said image; d) providing the bit-shifted image to a computer vision system; Including, The method, wherein the bit shifting includes adjusting the number of BPCs that each pixel reduces based on a feedback loop of steps a) through d) for each consecutive image in the set of consecutive images.

2. The method of claim 1 , wherein bit-shifting each pixel comprises reducing the total BPC for each pixel by a predetermined number of pixels.

3. 3. The method of claim 2, wherein bit-shifting each pixel comprises reducing the total BPC for each pixel from 8 BPC to 4 BPC.

4. 2. The method of claim 1, wherein the step of inverting the color data comprises performing a nonlinear logarithmic transformation on pixel bit-depth data using a log n(x) formula, where N is a log base value in the range of 2 to 10.

5. 2. The method of claim 1, wherein bit-shifting each pixel in the image comprises shifting image detail bits from most significant image bits to least significant image bits.

6. The method of claim 1 , wherein the bit-shifted image includes sufficient data elements to perform at least one process of the computer vision system.

7. The method of claim 1 , wherein the data elements include contrast lines and edge lines.

8. The method of claim 6 , wherein the bit-shifted image contains reduced color and / or pattern data.

9. At least one camera providing a video feed to the CMS controller A camera monitor system (CMS) for a vehicle, comprising: A camera monitor system, wherein the CMS controller includes a memory that stores instructions that cause the CMS controller to execute a video feed pre-processing method configured to invert color data of the video feed, then bit-shift each pixel in the video feed to reduce the total BPC (bits per color) of each pixel while maintaining image data elements, and provide the reduced BPC video feed to at least one computer vision system, wherein the resulting pixel size is controlled via a feedback loop including the bit-shifting process and a feedback output from the computer vision system.

10. 10. The camera monitor system of claim 9, wherein inverting the color data comprises performing a non-linear logarithmic transformation on pixel bit depth data.

11. 10. The camera monitor system of claim 9, wherein the video feed is an RGB888 video feed.

12. 12. The camera monitor system of claim 11, wherein bit-shifting each pixel in the video feed includes shifting the video feed from the RGB888 video feed to an RGB444 video feed.

13. bit-shifting each pixel in the video feed includes reducing the video feed from the RGB888 to an RGB feed having a pixel size of less than 8 bits; 12. The camera monitor system of claim 11, wherein the resulting pixel size is variable.

Images