Generating artificial color images from narrow spectral band data on camera-equipped vehicles

Abstract

A visual perception system includes a scanning camera, a color sensor having a color filter array (CFA), and a classifier node. The camera captures a full-color pixel image of a target object, such as a traffic signal light, and processes the pixel image through a narrow bandpass filter (BPF) such that the narrow BPF outputs a monochrome image of the target object. The color sensor and CFA receive the monochrome image. The color sensor has at least three color channels, each corresponding to a different color of spectral data in the monochrome image. The classifier node classifies constituent pixels of the monochrome image into different color bins as corresponding colors of interest using a predetermined classification decision tree. The colors of interest can be used to implement control actions, such as via an automatic driver assistance system (ADAS) control unit or indicator device.

Description

TECHNICAL FIELD

[0001] The present invention relates to a visual perception system, a motor vehicle comprising a visual perception system and a method for use on a motor vehicle. BACKGROUND

[0002] Perception camera systems commonly used on autonomous vehicles (AVs) are able to identify objects of interest through a large number of recognition parameters, including but not limited to the geometry, color and content of the objects. Such camera systems typically use RGB (red, green, blue) detectors / sensors to capture the full color field of the imaged scene. Narrow spectral band imaging systems specifically output a narrower portion of the available color spectrum of the scene. The resulting monochromatic pixel images thus complicate the use of such imaging systems in certain AV applications where accurate identification of the color of the objects in the imaged scene is required. However, narrow spectral band imaging camera systems offer performance advantages over traditional cameras, including enhanced scanning capabilities and the ability to output higher resolution images over a greater field of view.

[0003] Liquid crystal polarization grating (LCPG) devices are an example type of narrow bandpass imaging system that produces the above-mentioned monochromatic pixel images. The diffraction grating in LCPG devices has a narrow region in the spectral bandpass where the diffraction efficiency and overall transmission are maintained at relatively high values, typically greater than 70%. Furthermore, due to the common dispersion of the diffraction grating, a wide bandpass would produce a large dispersion effect, thus an undesirable dispersion effect, resulting in image blurring, which would severely limit the usefulness of such a device in certain AV applications. Therefore, in the type of scanning camera applications envisaged herein, it is highly desirable to use narrow spectral bandpass filters. SUMMARY

[0004] Disclosed herein are camera systems and related methods for artificially generating color images from the above-mentioned narrow spectral band data, for example on a motor vehicle. Generally speaking, the camera system comprises a scanning camera operable to collect a full color scene of potentially infinite color variations. The collected light is then passed through a narrow bandpass filter (BPF) after which the resulting light is sampled by a color sensor. This sampling process effectively separates the color information into three different color bins. Due to the narrow BPF, the generated image is predominantly monochromatic. The required colors are then classified based on the monochromatic pixel image, after first adjusting its parameters to artificially color the objects in the imaged scene, such as traffic lights, warning lights or similar hazard warnings. To obtain the best performance, the method described herein should be used in conjunction with a sensor that operates on quasi-monochromatic input, the function of which is to detect and locate traffic lights, warning signs, etc. as objects of interest to be colored. Quasi-monochromatic images can be generated by applying a learning filter with very limited spatial support (e.g. 3 x 3 or 5 x 5) to the raw data.

[0005] As part of the present method, the narrow BPF used here is located at the intersection between the two spectral filters commonly used in color sensors. Subsequently, starting from one exposure with two alternating color gain settings, multiple exposures with different durations are used, or another application suitable for a multi-level decision tree, in a typical three-color traffic light scenario, any pixel can present one of three predefined colors of interest, such as red, yellow, and green. With some modifications as described below, the present solution can be extended to one or more additional colors, such as in a four-color traffic light scenario with orange or amber as an additional color.

[0006] Aspects of the present disclosure include a vision perception system having a scanning camera, a color sensor, and a classifier node. The scanning camera is configured to capture a full-color pixel image of a target object and process the full-color pixel image through a narrow BPF such that the narrow BPF outputs a monochrome pixel image of the target object. The color sensor and accompanying color filter array (CFA) are in communication with the narrow BPF and configured to receive the monochrome pixel image therefrom. The color sensor has at least three color channels each corresponding to a different color of spectral data in the monochrome pixel image. The classifier classifies constituent pixels of the monochrome pixel image into one of a plurality of color bins using a predetermined classification decision tree as a corresponding color of interest.

[0007] In some embodiments, the image processing node artificially colors constituent pixels of the monochrome pixel image with the corresponding color of interest, thereby generating a color image.

[0008] The classifier node can be configured to determine a normalized digital value for each pixel of each RGB color channel and classify each constituent pixel of the monochrome pixel image into one of at least three colors of interest by comparing the normalized digital value to a scaled signal-to-noise ratio.

[0009] The color sensor in different exemplary embodiments is an RGB color sensor, a Bayer RGGB, or a YCCB sensor. The scanning camera can include a liquid crystal polarization grating.

[0010] The target object in a representative use case is a traffic light having a plurality of color bins including at least red, yellow, and green. The colors of interest in such an implementation include red, yellow, or green.

[0011] In some embodiments, the narrow BPF can have a bandwidth centered at approximately 600 nm.

[0012] The predetermined classification decision tree can operate by manipulating two independent gain settings for each color channel of a single exposure, the two independent gain settings having different color balance parameters. Alternatively, the predetermined classification decision tree can include processing multiple exposures of a full color pixel image, each of the multiple exposures having a different duration.

[0013] The visual perception system in some embodiments is configured to transmit an electronic signal indicative of the color of interest to an automated driver assistance system (ADAS) control unit of the motor vehicle. The visual perception system can also transmit an electronic signal indicative of the color of interest to the indicator device, thereby causing the indicator device to illuminate and / or broadcast an audible sound.

[0014] Also disclosed herein is a motor vehicle having a wheel connected to a vehicle body, an indicator device connected to the vehicle body, and a visual perception system connected to the vehicle body. In embodiments, a scanning camera of the visual perception system is configured to capture a full color pixel image of a multi-color traffic light and process the full color pixel image through a narrow BPF such that the narrow BPF outputs a single color pixel image of the multi-color traffic light.

[0015] The color sensor of the visual perception system is equipped with a color filter array (CFA). The color sensor and the CFA are in communication with and configured to receive the single color pixel image therefrom. The color sensor has at least three color channels, each color channel corresponding to a different color of spectral data in the single color pixel image. A classifier node is configured to classify constituent pixels of the single color pixel image into one of a plurality of color bins as a corresponding color of interest, the plurality of color bins including red, green, and yellow, wherein the classifier node is configured to output an electronic signal to the indicator device to activate the indicator device, the output signal indicative of the color of interest.

[0016] Also disclosed herein is a method for use on a motor vehicle having an indicator device and a visual perception system including a scanning camera. A possible implementation of the method is to capture a full color pixel image of a multi-color traffic light via the scanning camera and then process the full color pixel image through a narrow BPF such that the narrow BPF outputs a single color pixel image of the multi-color traffic light. The method further includes receiving the single color pixel image via a color sensor equipped with a color filter array (CFA). The color sensor has at least three color channels, each color channel corresponding to a different color of spectral data in the single color pixel image. The method of this embodiment includes classifying constituent pixels of the single color pixel image into one of a plurality of color bins as a corresponding color of interest with a classifier node, the plurality of color bins including red, green, and yellow, and then outputting an electronic signal to the indicator device to activate the indicator device, the electronic signal indicative of the color of interest.

[0017] The present invention comprises the following aspects.

[0018] Aspect 1. A visual perception system comprising:

[0019] a scanning camera configured to capture a panchromatic pixel image of a target object and process the panchromatic pixel image through a narrow band-pass filter (BPF) such that the narrow BPF outputs a monochromatic pixel image of the target object;

[0020] a color sensor equipped with a color filter array (CFA), wherein the color sensor and the CFA are in communication with the narrow BPF and configured to receive the monochromatic pixel image from the narrow BPF, and wherein the color sensor has at least three color channels each corresponding to a different color of spectral data in the monochromatic pixel image; and

[0021] a classifier node configured to classify constituent pixels of the monochromatic pixel image into one of a plurality of color bins as a corresponding color of interest using a predetermined classification decision tree.

[0022] Aspect 2. The visual perception system of Aspect 1, further comprising an image processing node configured to artificially color the constituent pixels of the monochromatic pixel image with the corresponding color of interest, thereby generating a color image.

[0023] Aspect 3. The visual perception system of Aspect 1, wherein the classifier node is configured to determine a normalized digital numerical value for each pixel of each RGB color channel and classify each of the constituent pixels of the monochromatic pixel image into one of at least three colors of interest by comparing the normalized digital numerical value to a scaled signal-to-noise ratio.

[0024] Aspect 4. The visual perception system of Aspect 1, wherein the color sensor is an RGB color sensor.

[0025] Aspect 5. The visual perception system of Aspect 1, wherein the color sensor is a Bayer RGGB or YCCB sensor.

[0026] Aspect 6. The visual perception system of Aspect 1, wherein the scanning camera comprises a liquid crystal polarization grating.

[0027] Aspect 7. The visual perception system of Aspect 1, wherein the target object is a traffic light, the plurality of color bins comprise at least red, yellow, and green, and the color of interest comprises one of red, yellow, or green.

[0028] Scheme 8. The visual perception system as described in Scheme 1, wherein the narrow BPF has a bandwidth centered at about 600 nm.

[0029] Scheme 9. The visual perception system as described in Scheme 1, wherein the predetermined classification decision tree includes operating two independent gain settings for each color channel of a single exposure, the two independent gain settings having different color balance parameters.

[0030] Scheme 10. The visual perception system as described in Scheme 1, wherein the predetermined classification decision tree includes processing multiple exposures of a full color pixel image, each exposure of the multiple exposures having a different duration.

[0031] Scheme 11. The visual perception system as described in Scheme 1, wherein the visual perception system is configured to send an electronic signal indicative of the color of interest to an automatic driver assistance system (ADAS) control unit of a motor vehicle.

[0032] Scheme 12. The visual perception system as described in Scheme 1, wherein the visual perception system is configured to send an electronic signal indicative of the color of interest to an indicator device, thereby causing the indicator device to emit light and / or broadcast an audible sound.

[0033] Scheme 13. A motor vehicle comprising:

[0034] a vehicle body;

[0035] wheels connected to the vehicle body;

[0036] an indicator device connected to the vehicle body; and

[0037] a visual perception system connected to the vehicle body and comprising:

[0038] a scanning camera configured to capture a full color pixel image of a multi-color traffic light and process the full color pixel image through a narrow band pass filter (BPF) such that the narrow BPF outputs a single color pixel image of the multi-color traffic light;

[0039] a color sensor equipped with a color filter array (CFA), wherein the color sensor and the CFA are in communication with and configured to receive the single color pixel image therefrom, and wherein the color sensor has at least three color channels each corresponding to a different color of spectral data in the single color pixel image; and

[0040] a classifier node configured to classify constituent pixels of the monochrome pixel image into one of a plurality of color bins as a corresponding color of interest using a predetermined classification decision tree, the plurality of color bins including red, green, and yellow, wherein the classifier node is configured to output an electronic signal to the indicator device to activate the indicator device, the output signal indicating the color of interest.

[0041] Scheme 14. The motor vehicle of Scheme 13, wherein the indicator device comprises a heads-up display.

[0042] Scheme 15. The motor vehicle of Scheme 13, further comprising an automated driver assistance system (ADAS) control unit operable to selectively control a dynamic state of the motor vehicle in response to the color of interest.

[0043] Scheme 16. The motor vehicle of Scheme 13, wherein the classifier node is configured to determine a normalized numerical value for each pixel of each RGB color channel, and classify each of the constituent pixels of the monochrome pixel image as one of at least three colors of interest by comparing the normalized numerical value to a scaled signal-to-noise ratio.

[0044] Scheme 17. The motor vehicle of Scheme 13, wherein the predetermined classification decision tree comprises (i) operating two independent gain settings for each color channel of a single exposure, the two independent gain settings having different color balance parameters, or (ii) processing multiple exposures of the full color image, each exposure of the multiple exposures having a different duration.

[0045] Scheme 18. A method for use on a motor vehicle having an indicator device and a vision perception system comprising a scanning camera, the method comprising:

[0046] capturing a full color pixel image of a multi-color traffic light via the scanning camera;

[0047] processing the full color pixel image through a narrow band pass filter (BPF) such that the narrow BPF outputs a monochrome pixel image of the multi-color traffic light;

[0048] receiving the monochrome pixel image via a color sensor equipped with a color filter array (CFA), wherein the color sensor has at least three color channels, each color channel corresponding to a different color of spectral data in the monochrome pixel image; and

[0049] classifying constituent pixels of the monochrome pixel image into one of a plurality of color bins as a corresponding color of interest using a classifier node, the plurality of color bins including red, green, and yellow; and

[0050] outputting an electronic signal to an indicator device to activate the indicator device, the electronic signal indicating the color of interest.

[0051] Scheme 19. The method as described in Scheme 18, further comprising selectively controlling a dynamic state of the motor vehicle via an automated driver assistance system (ADAS) control unit in response to the color of interest.

[0052] Scheme 20. The method as described in Scheme 18, further comprising calculating a normalized digital value for each pixel of each RGB color channel via the classifier node; and

[0053] classifying each constituent pixel of the monochromatic pixel image as one of at least three colors of interest, including comparing the normalized digital value to a scaled signal-to-noise ratio.

[0054] The foregoing features and advantages of the present disclosure, as well as other features and attendant advantages of particular embodiments thereof, will be more fully understood and appreciated by reference to the following detailed description, taken in conjunction with the accompanying drawings of which: BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a schematic diagram of a system for generating artificial color images from narrow spectral band data in accordance with the present disclosure, and a representative motor vehicle equipped with such a system.

[0056] Figure 2 is a plot of spectral filter responses and typical traffic signal spectra, with wavelength in nanometers (nm) shown on the horizontal axis, and relative response shown on the vertical axis.

[0057] Figure 3 is a three-dimensional vector plot illustrating the corresponding red, green, and blue (RGB) values representing the red, yellow, and green lights of a three-color traffic light.

[0058] Figure 4 is a schematic diagram of a classifier node that can be used as part of the system of Figure 1 .

[0059] Figure 5 and 6 are flowcharts describing methods of identifying light of a particular color for a three-color or four-color traffic light using the system of Figure 1 . DETAILED DESCRIPTION

[0060] The present disclosure can take form in various embodiments. Representative examples of the present disclosure are described in detail in the drawings and detailed description below. To the extent that the drawings and detailed description convey embodiments other than the ones described herein, they are intended to convey at least the spirit of the present teachings. As such, the abstract, the technical field, the summary, and the specific embodiments sections are not to be construed as limiting the scope of the claims.

[0061] For purposes of this specification, unless otherwise indicated, the use of the singular includes the plural, and vice versa, the term "and" and "or" shall be both conjunctive and disjunctive, the terms "any" and "all" shall both refer to "any and all," the terms "including," "containing," "comprising," "having," and the like shall be

[0062] Referring to the drawings, wherein like reference numbers refer to like features throughout the several views, Figure 1 A motor vehicle 10 is depicted, equipped with wheels 11 connected to a vehicle body 13. The motor vehicle 10 is depicted traveling along a surface 14 toward an intersection 16. In the example shown, traffic flow at the intersection 16 is regulated by a traffic light 18, which as shown has lenses 20L illuminated in red (R), yellow (Y), and green (G), respectively. Other colors can be present in other configurations of traffic lights, for example, four light embodiments that also include orange or amber.

[0063] For illustrative consistency and simplicity, an exemplary three-color embodiment of the traffic light 18 will be described below, without limiting the present teachings generally to traffic lights, or specifically to three-color traffic lights. Likewise, the motor vehicle 10 can be variously embodied as a crossover vehicle, a sport utility vehicle, a sedan, a truck, etc. Rail vehicles, boats, robots, transport equipment, motorcycles, etc. can also benefit from the present teachings, and thus, Figure 1 The motor vehicle 10 is but one possible host platform for the visual perception system 12.

[0064] The motor vehicle 10 includes a visual perception system 12. As contemplated herein, the visual perception system 12 is configured to add artificial color to a narrowband image of a target object, where the traffic light 18 represents such a target object. The present teachings aim to improve existing autonomous vehicle perception algorithms that exist with monochrome pixel image data. In monochrome pixel images, it can not be possible to accurately distinguish green light from yellow or red light, for example, due to the spectral content of the colors being fundamentally different. The color spectrum of a typical green, yellow, red three-color traffic light overlaps in the region of about 600 nm ± 15 nm, while only the tail of the green spectrum enters the same region. Thus, the relative response of a narrow BPF, such as the BPF 52 described below, can see a distinct difference between red and green, but little response in the blue spectrum, thereby minimizing the machine’s perception of green light. Thus, the present solution works in this region to facilitate automatic color recognition, and subsequent interaction with the motor vehicle 10 or its operator based on the improved color recognition capabilities enabled by the present teachings.

[0065] The visual perception system 12 as described herein includes a scanning camera 22 configured to image a target object, as represented by the wave 22W, in some configurations using a liquid crystal polarization grating by the scanning camera or a related image processing (IP) controller 50. The scanning camera 22 is in communication with the IP controller 50. The IP controller 50 as contemplated herein includes a color sensor 24, a classifier node 26, the narrow BPF 52 described above, and a color filter array 54. As understood in the art, the color filter array 54 includes a mosaic of color filters placed over the pixel sensors of the color sensor 24 and configured to capture color information. The particular embodiment that combines the color sensor 24 and the color filter array 54 can vary with the application, possible variations including a Bayer RGGB color filter, an RCCB, RCCG, RCCY, YCCB, RYYB, CYYM, or CYGM arrangement, where R, Y, and G stand for red, yellow, and green, and C stands for clear. While shown in Figure 1 While shown in FIG. 1 as being gathered together as constituent components of the IP controller 50 for ease and clarity of illustration, those skilled in the art will appreciate that the color sensor 24, the classifier node 26, the narrow BPF 52, and the color filter array 54 can be located at different locations on the motor vehicle 10, for example, within the scanning camera 22 or distributed as different processing nodes.

[0066] As described below with reference to FIG. 2, the color sensor 24 is configured to capture a color image of the target object, such as the traffic light 18, and the classifier node 26 is configured to classify the color image into a color class, such as a green, yellow, or red color class. The color filter array 54 is configured to filter the color image to produce a filtered color image, and the narrow BPF 52 is configured to pass a narrow band of wavelengths of the filtered color image to produce a narrowband image. The narrowband image is then used by the classifier node 26 to classify the color image into a color class. Figures 2-6In detail, the scanning camera 22 is configured to capture a full-color digital pixel image 25 of a target object, such as the region of interest 20 of the traffic light 18. The pixel image 25 is then processed by the narrow BPF 52 such that the narrow BPF 52 ultimately outputs a monochromatic pixel image 27 of the imaged target object. The color sensor 24, in turn, is equipped with a color filter array 54, the color sensor 24 and color filter array 54 being in communication with the narrow BPF 52 and thus configured to receive the monochromatic pixel image 27 therefrom. The color sensor 24 has three or more color channels, each corresponding to a different color of the spectral data in the monochromatic pixel image 27, i.e., red, green, or yellow in the example three-color configuration of the traffic light 18.

[0067] Further to the visual perception system 12 of Figure 1 The classifier node 26 is configured to classify the constituent pixels of the monochromatic pixel image 27 into one of a plurality of different color categories or "bins" as a corresponding color of interest using a predetermined classification decision tree and prior training as described below. Two or more color bins are contemplated herein to provide a basic binary classification. For increased capability, the classifier node 26 can be trained to classify the constituent pixels of the monochromatic pixel image 27 into three or more color bins.

[0068] The visual perception system 12 can be used on the motor vehicle 10 to provide a number of performance advantages. By way of illustrative example of this, the motor vehicle 10 can be equipped with an advanced driver assistance system (ADAS) control module 30, the ADAS control unit 30 being in turn operable to automatically dynamically control the motor vehicle 10 in response to the electronic signal (arrow 300) output by the IP controller 50. As known in the art, related ADAS functionality can include performing braking, steering, and / or acceleration operations of the motor vehicle 10 based on the output of the visual perception system 12. For example, the ADAS control unit 30 can include an automatic braking system control module operable to slow or stop the motor vehicle 10 in response to a detected road hazard, as well as other automatic systems such as automatic lane changing, lane keeping, steering, collision avoidance, adaptive cruise control, etc. To this end, the visual perception system 12 is also capable of informing an indicator device 32 (lights 32L and / or speakers 32S) of a detected color of the traffic light 18 to a driver of the motor vehicle 10 via a driver alert signal (arrow 31). Thus, the visual perception system 12 can be configured to send an electronic signal, e.g., a driver alert signal, indicating the color of interest to illuminate and / or play an audible sound.

[0069] Continuing with the description of the visual perception system 12 of Figure 1The scanning camera 22 is configured to capture a panchromatic image of the region of interest 20 of the traffic light 18 in an exemplary use case, in which the visual perception system 12 is used to detect whether the traffic light 18 is currently green, yellow, or red. Typical embodiments may utilize complementary metal-oxide-semiconductor (CMOS) sensors, charge-coupled device (CCD) sensors, or other suitable optical sensors to generate images indicating the field of view of the motor vehicle 10. Such devices may be configured to continuously generate images, for example, tens of images per second. Figure 1 In the scenario shown, the target object is or includes traffic light 18, while in other applications, the target object may include, for example, a hazard light, a traffic control sign, or other lights, signs, or other objects with multiple different colors indicating the corresponding information.

[0070] The narrow BPF 52 mentioned above is configured to receive a panchromatic image from the scanning camera 22 (arrow 25) and output a monochrome pixel image of the target object (arrow 27), as briefly mentioned above. The color sensor 24, in turn, communicates with the narrow BPF 52, and is operable to generate three-dimensional (3D) color vectors, such as... Figure 3 As shown, it indicates the spectral content of the monochrome pixel image (arrow 27). Classifier node 26 is configured to adjust the parameters of the aforementioned 3D color vector to derive artificial color data, and then classify the artificial color data into a color bin as the color of interest.

[0071] To ensure that the IP controller 50 can implement this quickly and accurately, please refer to the following reference. Figures 2-6 The IP controller 50, which details various image processing functions, can be equipped with volatile and non-volatile memory (M) 57, one or more processors (P) 58, and associated hardware such as digital clocks or oscillators, input / output circuitry, buffer circuitry, application-specific integrated circuits (ASICs), system-on-a-chip (SoCs), electronic circuitry, and other necessary hardware to provide programmable functionality. In the context of this disclosure, the electronic control unit can execute instructions via the processor to cause the IP controller 50 to perform... Figure 5 and Figure 6 This method 100 and / or 100A receives and processes various data, as described below. Figure 5 and Figure 6 An embodiment thereof is described. The IP controller 50 may also include an optional image processing node 55 configured to artificially color the constituent pixels of the monochrome pixel image 27 with corresponding colors of interest to generate a color image, for example, for display on a HUD 32H.

[0072] This solution adds artificial colors to narrowband images of traffic lights to improve the AV perception algorithm of visual perception systems. In monochrome images, people may not be able to distinguish green light from yellow or red light due to differences in spectral content. The spectra of a typical three-color traffic light with green, yellow, orange, and red lights overlap in the range of 600nm ± 15nm, with only the tail end of the green spectrum entering this same range. Therefore, the relative response of a narrow BPF in a PCPG may show a significant difference between red and green, with almost no response in the blue spectrum, thus minimizing the perception of green light.

[0073] refer to Figure 2 Curve 40 illustrates a typical spectral filter response 152 for a narrow BPF 52, used for representative green, yellow, and red traffic light spectra, with specific wavelengths (λ) in nanometers (nm) and relative responses (RR) plotted on the corresponding horizontal and vertical axes. The illustrated filter response 152 passes light in a predetermined frequency band centered at approximately 600 nm, or from 580 nm to 620 nm in the illustrated non-limiting embodiment. Traces 44Y, 44R, and 44G show the corresponding spectral data for yellow, red, and green traffic lights, respectively. In this particular example, the spectral bandpass of the narrow BPF 52 centered at 600 nm has a full width at half maximum (FWHM) of approximately 30 nm.

[0074] A narrow BPF 52, for example, when coupled to a CMOS or CD sensor with a standard Bayer filter pattern, will produce a partially color image with minimal contribution to the blue gamut. Red and yellow traffic lights are still easily detected because the spectral bandwidth of such light sources overlaps with the spectral region of the color sensor. However, the green spectrum peaks outside the bandwidth of the narrow BPF 52 at approximately 515 nm, so a sufficiently high response above the system noise is not recorded to capture a reliable signal from the blue and green channels.

[0075] Now for reference Figure 3 Within the scope of this disclosure, the problem solved by the visual perception system 12 can be categorized as an N-class classification problem. Figure 1 In an exemplary case of a three-color traffic light 18, where N=3 (red, yellow, and green), incident light passing through the narrow BPF 52 and the color filter array 54 of the color sensor 24 is encoded as a 3D vector representing collective RGB values. Due to the diversity of light sources, ambient lighting conditions, and camera settings, the RGB values ​​(Figure 4) and... Figure 3The specific mapping between the red, yellow, and green classes 60 is called a distribution. As a simple approximation, each of the three classes in this example can be assumed to be a Gaussian distribution centered at a specific RGB value, with a certain variance. When representing RGB vectors in 3D space, the "correct" color class can be determined by... Figure 1 The IP controller 50 determines this by finding separate hyperplanes 61, 62, and 63 between the color clusters.

[0076] like Figure 4 As shown, classifier node (NC) 26 receives the aforementioned RGB vector, represented by its constituent color values ​​R, G, and B (arrows 64, 65, and 66, respectively), and outputs a color estimate (arrow 68) that maximizes the probability conditioned on the RGB input. The narrow BPF52 significantly reduces the "blue" signal entering classifier node 54. Nevertheless, a small residual blue signal enables robust separation of three or more colors. To this end, classifier node 26 adjusts the parameters of the RGB vector to derive the artificial color data, using one of several possible alternative methods described below. Subsequently, classifier node 26 classifies the artificial color data into different color bins as colors of interest, for example... Figure 1 Red, yellow, or green light in non-restricted tri-color traffic light use cases.

[0077] Classifier Node Functionality: Continuing with the example of the three-color traffic light 18, we can assume a simple Gaussian distribution for the three colors. Therefore, for the purposes of this solution, it is sufficient to sample representative images covering a wide variety of light across the spectrum under various ambient lighting conditions. For example, classifier node 26 can utilize different times of day, night, during inclement weather, etc. Figure 1 The traffic light 18 is trained using sample images of reference red, yellow, and green light with the same or similar spectra. The IP controller 50 can find the mean ( μ i ) and variance ( σ i 2 ) or each color ( i Classifier node 26 calculates a specific color as follows ( i The probability of ) is:

[0078] Where x == RGB input signal, i = [red, yellow, green], and predict color ( i )= Arg max i Prob(i|x).

[0079] here, x , μ and σ2 It is a 3D vector. The index "i" represents a specific color. If the reduced frequency response available after using a narrow BPF 52 is distinguishable, the number of classes (i.e., colors) can be increased. When a simple Gaussian distribution model is insufficient, supervised learning-based N-class classification schemes such as, but not limited to, logistic regression, multi-class support vector machines (SVM), decision trees, and neural networks can be used. In the learning-based method, a representative set of images can be used to train classifier nodes 26 and identify their optimal parameters. Within the scope of this disclosure, this learning-based method is likely optimal for deriving artificial color rendering.

[0080] Independent Gain Settings: This approach implements the teachings by applying two independent gain settings (digital or analog) to the captured RGB data, with different color balance parameters. For example, the relative gain (amplification) for each color channel can be set according to a table for each frame:

[0081]

[0082] Find the maximum number of numbers per exposure (D) n (For reference only) Figure 1 The maximum D of a representative 8-bit embodiment of the color sensor 24 n The value is 256. Then, for each color channel, the D value for each pixel is found. nR,G,B The value is calculated, and the IP controller 50 calculates the normalized value, i.e.:

[0083]

[0084] This process generates six values ​​from two gain sets and three RGB values, for example Figure 4 Arrows 64, 65, and 66, then... Figure 5 Method 100 uses these values ​​to determine the color of the imaged target object.

[0085] This assumes that the noise of the camera system has been measured as part of the normal calibration process and assigned a σ value. Therefore, the scene signal-to-noise ratio (SNR) is calculated for the purposes of this derivation as follows: The maximum signal is taken from the imaging scene. This value is then scaled using a scaling parameter (P1) determined during calibration. A single value for the scaling parameter (P1) is determined by calibrating the camera against a reference light source that conforms to the expected spectral distribution of such devices, used in standard traffic lights. Since P1 is a scaling factor, its value can be determined by running the algorithm multiple times until convergence is proven. Therefore, the scaled SNR is expressed as... .

[0086] Now for reference Figure 5 Provides identificationFigure 1 an exemplary embodiment of the method 100 of classifying the red, yellow, and green lights of a three-color traffic light 18, wherein Figure 5 The classification function of the classifier node 26 is described. It is assumed here that the traffic light 18 can be identified by other conventional means, such as object recognition, in order to allow the method 100 to continue in the region of interest of the traffic light 18 without worrying about surrounding colors in the imaged scene. Those skilled in the art will appreciate that the method 100 can be used to identify other colors, so that red, yellow, and green are merely illustrative of one application of the present teachings. Figure 5 The flowchart representation of the decision process via the above-described criteria begins at block B102, generating pixel-level red, green, and blue (RGB) data, i.e., as described above Figure 4 by the arrows 64, 65, and 66. The collective RGB data can be adjusted by, for example, the two color gains described above, by multiple exposures of different durations, or using other decision trees in different embodiments. Block B102 produces values and SNR as described above. The method 100 then proceeds to block B104.

[0087] Block B104 requires comparing the above-described values to each other, for example, using a comparator circuit of the IP controller 50. If the SNR is exceeded, and at the same time the SNR is less than, the classifier node 26 proceeds to block B105. Otherwise the method 100 proceeds to block B106.

[0088] At block B105, the IP controller 50 can record an alphanumeric code in memory, which in this example indicates that the pixel (Pxl) color is yellow.

[0089] Block B106 implements a different comparison in response to the negative decision of block B104. Here, the classifier node 26 determines whether the two normalized values are equal to zero, and whether the normalized value is less than the scaled signal-to-noise ratio, i.e., SNR. When both conditions are true, the method 100 proceeds to block B107, and blocks B109 in the alternative.

[0090] Figure 5 Block B107 includes recording an alphanumeric code in memory 57, which indicates that the pixel color is green. Thus, block B107 is similar to block B105.

[0091] Block B109 includes recording an alphanumeric code in memory 57, which indicates that the pixel color is red. Thus, block B109 is similar to block B105, similarly to block B107.

[0092] Those skilled in the art will understand that additional colors can be categorized according to this teaching. Figure 5 The flowchart can be slightly modified. For example, such as Figure 6 The method 100A shown implements four-color classification, which can be implemented using a four-color traffic light with red, green, yellow, and orange / amber lights. For this purpose, method 100A uses an additional measurement point. This point can be determined as the ratio between the green and red values, i.e. This additional parameter helps. Figure 1 The IP controller 50 distinguishes between yellow and orange light spectra based on the difference in the amount of green light in the yellow and orange light. This value is compared with another calibration parameter P2, which is determined during calibration through iteration with the orange light source, specifically during the training of classifier node 26.

[0093] From similar Figure 4 Starting with block B102A of block B102, pixel-level RGB data is determined via the ECU. In this case, the RGB data will include... SNRS, Zrg, and P2. Once determined, method 100A proceeds to block B104A.

[0094] In block B104A, IP controller 50 next determines whether... Exceeding SNR If the SNR is less than the calibration value P2, Zrg exceeds the calibration value P2. In this case, method 100A proceeds to block B105A, and in the alternative method, it proceeds to block B106A.

[0095] At block B105A, which is similar to block B105, IP controller 50 registers an alphanumeric code in memory 57 indicating that the pixel color is yellow.

[0096] Block B106A is similar to block B106, and similarly includes determining whether there are two normalized values. Equal to zero, and whether the value is normalized. Less than the scaled signal-to-noise ratio (SNR). When both conditions are true, method 100A proceeds to block B107A, and in the alternative, it proceeds to block B108.

[0097] Block B107A includes an alphanumeric code registered in memory 57 indicating that the pixel color is green, similar to... Figure 4 Block B107.

[0098] At block B108, the IP controller 50 again evaluates the parameters Zrg and P2. Block B108 is similar to block 104A, except that it verifies that Zrg is less than P2. When the condition of block B108 has been met, the method 100A proceeds to block Bll l and, in the alternative, to block B109A.

[0099] Block B109A requires the registration in the memory 57 of an alphanumeric code indicating that the color of the pixel is red.

[0100] Block Bll l requires the registration in the memory 57 of an alphanumeric code indicating that the color of the pixel is orange.

[0101] Red light: The present solution can be understood with reference to the working example. A red traffic light having representative RGB values of 255, 31, 3, i.e. R = 255, G = 31 and B = 3, is imaged using the RGB sensor 32R. nmax For two exposures under the above-mentioned representative gain settings, the resulting average D n values are:

[0102]

[0103] The subsequent six values are calculated as:

[0104]

[0105] According to the flowchart of Figure 4 , with an exemplary SNR value of 0.072, the IP controller 50 will flow from block B102 to block B104, to block B106, to block B109. Thus, the specific color returned from this exemplary data set will be red. Therefore, an object similar to a traffic light and having the above-mentioned exemplary RGB values can be artificially "colored" red in the monochrome image generated by the RGB sensor, whether in the logic of the IP controller 50 or in the image actually displayed. In embodiments, Figure 1 The ADAS control unit 30 of Figure 1 may command a braking response of the motor vehicle 10 as needed, for example, by adjusting the braking force based on the operator's braking response to the red light, the vehicle speed trajectory, etc., or can simply alert the operator of the presence of the red light through an audible, visual and / or tactile alarm, a heads-up display (HUD) 32H, etc.

[0106] Yellow light: In another example, a yellow traffic light can be imaged using representative RGB values of 255, 213 and 4. For two exposures under the above-mentioned gain settings, the resulting average D n values are:

[0107]

[0108] Subsequent The six values of

[0109]

[0110] According to the flowchart of Figure 4 In the case of an exemplary SNR of 0.072, the ECU will flow from block B102 to block B104, to block B105. Thus, the color returned from this exemplary data set will be yellow. Therefore, if an object similar to traffic light 18 having the above-mentioned RGB values is identified in the collected image, it can be artificially "colored" yellow in the monochrome pixel image generated by the color sensor. In a possible embodiment, Figure 1 The ADAS control unit 30 of

[0111] Green light: In yet another example of the present teachings, a green traffic light can be imaged with representative RGB values of 22, 248, and 164. For two exposures under the gain settings described previously, the resulting average D n values are:

[0112]

[0113] Subsequent The six values of

[0114]

[0115] According to the flowchart of Figure 4 In the case of an exemplary SNR of 0.072, the ECU will flow from block B102 to block B104, to block B106, to block B107. Thus, the color returned from this exemplary data set will be green. Therefore, an imaged target object similar to traffic light 18 having the above-mentioned RGB values can be artificially "colored" green in the monochrome pixel image 27 generated by the color sensor 24. Figure 1

[0116] In Figure 5 and Figure 6 ​In the methods 100 and 100A, the artificial coloring of the pixels can occur before demosaicing and before gamma correction, or the method can avoid the use of a demosaicing process altogether. As understood in the art, typical demosaicing algorithms are not expected to perform well on raw images after BPF. Thus, one working within the scope of the disclosed methods 100 and 100A can separately group the R, G, and B pixels that fall within a given region of interest to be colored. The described classification strategy will then operate on these pixel sets. Additionally, if the color filter array 54 is not of the RGB type, then an appropriate color transform should be applied to the output of the color filter array 54 so that the described methods 100 and 100A will operate in the RGB space as expected.

[0117] In some embodiments, aspects of the present disclosure can be implemented by way of a computer-executable program of instructions, such as a software application or an application programmed to perform a function by any of the controllers or controller variants described herein. In non-limiting examples, the software can include routines, programs, objects, components, and data structures that perform particular tasks or implement particular data types. The software can form an interface to allow a computer to react to a source of input. The software can also cooperate with other code segments to initiate a variety of tasks in response to data received in association with the source of the input. The software can be stored on any of a variety of memory media, such as CD-ROM, diskette, and semiconductor memory (e.g., various types of RAM or ROM).

[0118] Furthermore, aspects of the present disclosure can be practiced with various computer system and computer network configurations, including a multiprocessor system, a microprocessor-based or programmable consumer electronics, a minicomputer, a mainframe computer, and the like. Additionally, aspects of the present disclosure can be practiced in distributed computing environments where tasks are performed by local and remote processing devices that are linked (either by hardwired links, wireless links, or by a combination thereof) through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices. As such, aspects of the present disclosure can be implemented in connection with various hardware, software, or combinations thereof, in a computer system or other processing system.

[0119] Any of the methods described herein can include machine readable instructions for execution by: (a) a processor, such as processor 58, (b) a controller, such as IP controller 50, and / or (c) another suitable processing means. The algorithms, software, control logic, protocols or methods disclosed herein can be embodied in software stored on a tangible media such as flash memory, solid state memory (SSD) memory, hard disk drive (HDD) memory, CD-ROM, digital versatile disk (DVD), or other memory devices. The entire algorithm, control logic, protocol or method, and / or portions thereof, can alternatively be embodied in hardware, and / or in a firmware that is usefully tangibly embodied in a hardware read-only memory (ROM), such as flash memory, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), hard disk drive, or other memory device. The entire algorithm, control logic, protocol or method, and / or portions thereof, can alternatively be embodied in a combination of software and hardware. Further, although specific algorithms can be described herein with reference to flow charts and / or work flow diagrams, many other methods of implementing example machine readable instructions can be used. Moreover, the instructions embodying method 100 or 100A can be recorded on non-transitory computer readable storage media. These instructions, when executed by processor 58, cause IP controller 50 to perform the processing described above.

[0120] The detailed description and accompanying drawings or diagrams provide support for the teachings described herein, but the scope of the teachings is defined solely by the claims. While several best modes and other embodiments of the teachings have been described in detail, various alternative designs and embodiments exist for practicing the teachings defined in the appended claims. Furthermore, the disclosed disclosure expressly encompasses combinations and sub-combinations of the elements and features presented above and below.

Claims

1. A visual perception system comprising: a liquid crystal polarization grating scanning camera configured to capture a full color pixel image of a target object and process the full color pixel image through a narrow bandpass filter such that the narrow bandpass filter outputs a monochromatic pixel image of the target object, wherein the narrow bandpass filter is centered at 600 nm, has a bandwidth of 40 nm, and a full width at half maximum of 30 nm; a color sensor equipped with a color filter array, wherein the color sensor and the color filter array are configured to receive the monochromatic pixel image from the narrow bandpass filter, and wherein the color sensor has at least three color channels each corresponding to a different color of spectral data in the monochromatic pixel image; and a classifier node configured to classify a constituent pixel of the monochromatic pixel image into one of a plurality of color bins as a corresponding color of interest using a predetermined classification decision tree, including computing a respective probability of each of the constituent pixel being red, yellow, or green as the corresponding color of interest, and outputting a color estimate having a greatest probability.

2. The visual perception system of claim 1, further comprising an image processing node configured to artificially color the constituent pixel of the monochromatic pixel image with the corresponding color of interest, thereby generating a color image.

3. The visual perception system of claim 1, wherein the classifier node is configured to determine a normalized digital number value for each pixel of each RGB color channel and classify each of the constituent pixel of the monochromatic pixel image into one of the red, yellow, or green colors by comparing the normalized digital number value to a scaled signal-to-noise ratio.

4. The visual perception system of claim 1, wherein the color sensor comprises an RGB color sensor.

5. The visual perception system of claim 1, wherein the color sensor comprises a Bayer RGGB or YCCB sensor.

6. The visual perception system of claim 1, wherein the target object is a traffic light, the plurality of color bins includes at least red, yellow, and green, and the color of interest includes one of red, yellow, or green.

7. The visual perception system of claim 1, wherein the predetermined classification decision tree includes operating two independent gain settings for each color channel of a single exposure, the two independent gain settings having different color balance parameters.

8. The visual perception system of claim 1, wherein the predetermined classification decision tree includes processing multiple exposures of a full color pixel image, each exposure of the multiple exposures having a different duration.

9. The visual perception system of claim 1, wherein the visual perception system is configured to send an electronic signal indicative of the color of interest to an automated driver assistance system (ADAS) control unit of a motor vehicle.

10. The visual perception system of claim 1, wherein the visual perception system is configured to send an electronic signal to an indicator device indicating the color of interest, thereby causing the indicator device to emit light and / or broadcast an audible sound.

11. A motor vehicle comprising: a vehicle body; wheels connected to the vehicle body; an indicator device connected to the vehicle body; and a visual perception system connected to the vehicle body and comprising: a liquid crystal polarization grating scanning camera configured to capture a full color pixel image of a multi-color traffic light and process the full color pixel image through a narrow bandpass filter such that the narrow bandpass filter outputs a single color pixel image of the multi-color traffic light, wherein the narrow bandpass filter is centered at 600 nm with a bandwidth of 40 nm and a full width at half maximum of 30 nm; a color sensor equipped with a color filter array, wherein the color sensor and the color filter array are configured to receive the single color pixel image from the narrow bandpass filter, and wherein the color sensor has at least three color channels each corresponding to a different color of spectral data in the single color pixel image; and a classifier node configured to classify a constituent pixel of the single color pixel image into one of a plurality of color bins as a corresponding color of interest using a predetermined classification decision tree, including computing a respective probability of each of the constituent pixel being red, yellow, or green as the corresponding color of interest, and outputting a color estimate having a maximum probability, the plurality of color bins including red, green, and yellow, wherein the classifier node is configured to output an electronic signal to the indicator device to activate the indicator device, the output signal indicating the color of interest.

12. The motor vehicle of claim 11, wherein the indicator device comprises a heads-up display.

13. The motor vehicle of claim 11, further comprising an automatic driver assistance system (ADAS) control unit operable to selectively control a dynamic state of the motor vehicle in response to the color of interest.

14. The motor vehicle of claim 11, wherein the classifier node is configured to determine a normalized digital numerical value for each pixel of each RGB color channel, and classify each of the constituent pixels of the single color pixel image into one of at least three colors of interest by comparing the normalized digital numerical value to a scaled signal-to-noise ratio.

15. The motor vehicle of claim 11, wherein the predetermined classification decision tree includes (i) operating two independent gain settings for each color channel of a single exposure, the two independent gain settings having different color balance parameters, or (ii) processing a plurality of exposures of the full color pixel image, each exposure of the plurality of exposures having a different duration.

16. A method for use on a motor vehicle having an indicator device and a visual perception system comprising a liquid crystal polarization grating scanning camera, the method comprising: ​ capturing a full color pixel image of a multi-color traffic light via the liquid crystal polarization grating scanning camera; processing the full color pixel image through a narrow band pass filter such that the narrow band pass filter outputs a single color pixel image of the multi-color traffic light, wherein the narrow band pass filter is centered at 600 nm with a 40 nm bandwidth and a 30 nm full width at half maximum; receiving the single color pixel image via a color sensor equipped with a color filter array, wherein the color sensor has at least three color channels each corresponding to a different color of spectral data in the single color pixel image; and classifying constituent pixels of the single color pixel image into one of a plurality of color bins as a corresponding color of interest using a classifier node, the plurality of color bins including red, green, and yellow, including calculating a respective probability of each of the constituent pixels as the corresponding color of interest being red, yellow, or green; and outputting an electronic signal to an indicator device to activate the indicator device, the electronic signal indicating the color of interest, wherein the color of interest is the color estimate with the greatest probability.

17. The method of claim 16, further comprising selectively controlling a dynamic state of the motor vehicle in response to the color of interest via an automated driver assistance system (ADAS) control unit.

18. The method of claim 16, further comprising calculating a normalized digital numerical value for each pixel of each RGB color channel via the classifier node; and classifying each of the constituent pixels of the single color pixel image into one of at least three colors of interest, including comparing the normalized digital numerical value to a scaled signal-to-noise ratio.

19. The method of claim 16, wherein the color of interest is one of red, yellow, and green.

Images