Color Learning

Abstract

A computing device is programmed to acquire a color image and transform the color image into a color component map. The computer may be further programmed to process the color component map to detect a traffic sign by determining spatial correspondence between the color component map and the traffic sign and determining temporal consistency.

Description

Technical Field

[0001] The disclosure relates generally to the field of autonomous or semi-autonomous vehicle operation, and more particularly to methods and systems for color learning, for example, to recognize traffic signs. Background Art

[0002] The vehicle may be equipped to operate in both an autonomous mode and an occupant driving mode. The vehicle may be equipped with computing devices, networks, sensors, and controllers to acquire information about the vehicle's environment and to drive the vehicle based on the information. Safe and comfortable driving of the vehicle may depend on acquiring accurate and timely information about the vehicle's environment. The computing devices, networks, sensors, and controllers may be equipped to analyze their performance; detect when information is not acquired in an accurate and timely manner; and take corrective action, including notifying an occupant of the vehicle, relinquishing autonomous control, or stopping the vehicle. Summary of the invention

[0003] The vehicle may be equipped to operate in both an autonomous mode and an occupant-driven mode. By semi-autonomous mode or fully autonomous mode, we mean an operating mode in which the vehicle may be driven by a computing device that is part of a vehicle information system having sensors and controllers. The vehicle may be occupied or unoccupied, but in either case the vehicle may be driven without occupant assistance. For purposes of this disclosure, an autonomous mode is defined as a mode in which each of vehicle propulsion (e.g., via a powertrain including an internal combustion engine and / or an electric motor), braking, and steering is controlled by one or more vehicle computers; in a semi-autonomous mode, the vehicle computers control one or both of vehicle propulsion, braking, and steering.

[0004] Disclosed herein is a method comprising: acquiring a color image; transforming the color image into a color-component map; and processing the color-component map to detect a traffic sign by determining spatial correspondence and temporal consistency between the color-component map and the traffic sign. The color image may be transformed into the color-component map by converting a three-element RGB vector representing a pixel of the color image into an eleven-element color name vector according to an RGB to color name space transformation determined based on training data. Each eleven-element color name vector may represent a single color. A color basis may be generated using training data by K-means clustering, the training data determining the probability of occurrence of the eleven-element color name vector, and the training data may include an image containing a traffic sign. The color component map may be processed by segmenting the color component map using thresholding. Processing the color component map may be based on spatial correspondence, wherein the segmented color components are retained based on matching the color components included in the traffic sign, while spatially nearby color components match the color components included in the traffic sign.

[0005] A second color image may be acquired, and the second color image may be transformed into a second color component map, the second color component map may be segmented, and spatial correspondence and temporal consistency determined in the second color component map may be determined based on comparing the color component map from the second image with the color component map from the first image. Determining temporal consistency may preserve color components that appear in traffic signs in two or more color images acquired at different times. Traffic signs may be detected based on determining optical flow, wherein the optical flow predicts where traffic signs appear based on information related to sensor movement and camera lenses. Training may include performing a k-cluster analysis on color name vectors. Thresholding the color component map may be based on an Otsu thresholding technique, and the Otsu thresholding technique may produce a binary output image, wherein the binary output image is combined with a color image to produce an output color image.

[0006] Further disclosed is a computer readable medium storing program instructions to perform some or all of the above method steps. Further disclosed is a computer programmed to perform some or all of the above method steps, the computer comprising a computer device programmed to: acquire a color image; transform the color image into a color component map; and process the color component map to detect the traffic sign by determining spatial correspondence and temporal consistency between the color component map and the traffic sign. The computer may be further programmed to transform the color image into a color component map to indicate the significant colors of the traffic sign by: transforming a three-element RGB vector representing a pixel of the color image into an eleven-element color name vector according to an RGB to color name transformation determined based on training data, wherein each eleven-element color name vector may represent a single color as an output of the color space transformation. The computer may be further programmed to determine a probability based on the training data indicating a probability of occurrence of the eleven-element color name vector, and the training data may include an image containing a traffic sign. The color component map may be processed by segmenting the color component map using thresholding. Processing the color component map may be based on spatial correspondence, wherein segmented color components are retained based on matching color components included in a traffic sign, while spatially nearby color components match color components included in the traffic sign.

[0007] The computer may be further programmed to acquire a second color image, and may transform the second color image into a second color component map, may segment the second color component map, and may determine spatial correspondence and temporal consistency determined in the second color component map based on comparing the color component map from the second image with the color component map from the first image. Determining temporal consistency may preserve color components that appear in traffic signs in two or more color images acquired at different times. The computer may be further programmed to detect traffic signs based on determining optical flow, wherein the optical flow predicts where traffic signs appear based on information related to sensor movement and camera lenses. Training may include performing a k-cluster analysis on color name vectors. Thresholding the color component map may be based on an Otsu thresholding technique, and the Otsu thresholding technique may produce a binary output image, wherein the binary output image is combined with a color image to produce an output color image. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a block diagram of an exemplary vehicle.

[0009] Figure 2 is a diagram of an exemplary video image including a traffic sign.

[0010] Figure 3 is a diagram of an exemplary color space image encoded as three RGB image planes.

[0011] Figure 4 is a diagram of an exemplary color space conversion from RGB to a color name space.

[0012] Figure 5 is a diagram of three exemplary color component maps.

[0013] Figure 6 is a diagram of an exemplary spatial conformance process applied to a color component map.

[0014] Figure 7 is a diagram of an exemplary temporal consistency process applied to a color component map.

[0015] Figure 8 is a diagram of an exemplary process for generating a color component map.

[0016] Fig. 9 is a flow chart of an exemplary process for detecting traffic signs using color processing.

[0017] Fig.10 is a flow chart of an exemplary process for training color component clusters.

[0018] Fig.11 is a flow chart of an exemplary process for detecting traffic signs by determining spatial correspondence.

[0019] Fig.12 is a flow chart of an exemplary process for detecting traffic signs by determining temporal consistency. DETAILED DESCRIPTION

[0020] Figure 1 1 is a diagram of a vehicle information system 100 according to a disclosed implementation that includes a vehicle 110 that can operate in an autonomous (“autonomous” automatically refers to “fully autonomous” in this disclosure) mode and an occupant-driven (also referred to as non-autonomous) mode. The vehicle 110 also includes one or more computing devices 115 to perform calculations for driving the vehicle 110 during autonomous operation. The computing device 115 can receive information about the operation of the vehicle from sensors 116.

[0021] The computing device 115 includes, for example, a known processor and memory. In addition, the memory includes one or more forms of computer-readable media and stores instructions that can be executed by the processor to perform various operations (including the operations disclosed herein). For example, the computing device 115 may include a program to operate one or more of vehicle braking, propulsion (e.g., controlling acceleration in the vehicle 110 by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior lights and / or exterior lights, etc., and determine whether and when the computing device 115 (as opposed to a human operator) controls such operations.

[0022] The computing device 115 may include or be communicatively coupled to one or more computing devices (e.g., via a vehicle communication bus further described below), such as controllers included in the vehicle 110 for monitoring and / or controlling various vehicle components, such as a powertrain controller 112, a brake controller 113, a steering controller 114, etc. The computing device 115 is generally arranged to communicate on a vehicle communication network, such as a bus (e.g., a controller area network (CAN), etc.) in the vehicle 110; the vehicle 110 network may include, for example, known wired or wireless communication mechanisms, such as Ethernet or other communication protocols.

[0023] The computing device 115 may transmit messages to various devices in the vehicle via the vehicle network, and / or receive messages from various devices (e.g., controllers, actuators, sensors, etc.) (including sensor 116). Alternatively, or in addition, where the computing device 115 actually includes multiple devices, the vehicle communication network may be used for communication between devices represented as computing device 115 in this disclosure. In addition, as mentioned below, various controllers or sensing elements may provide data to the computing device 115 via the vehicle communication network.

[0024] Additionally, the computing device 115 may be configured to communicate with a remote server computer 120 (e.g., a cloud server) via a network 130 through a vehicle-to-infrastructure (V-to-I) interface 111, as described below, which may utilize various wired and / or wireless networking technologies, such as cellular, and wired and / or wireless packet networks. The computing device 115 may be configured to communicate with other vehicles 110 through the V to I interface 111 using a vehicle-to-vehicle (V to V) network formed specifically between nearby vehicles 110 or through an infrastructure-based network. The computing device 115 also includes, for example, a known non-volatile memory. The computing device 115 may record the information by storing it in the non-volatile memory for later retrieval and transmission to the server computer 120 or the user mobile device 160 via the vehicle communication network and the vehicle-to-infrastructure (V to I) interface 111.

[0025] As already mentioned, the program is generally included in instructions that are stored in memory and executed by the processor of the computing device 115 to operate one or more vehicle 110 components, such as braking, steering, propulsion, etc., without the intervention of a human operator. The computing device 115 can make various decisions and / or control various vehicle 110 components and / or operations without a driver to operate the vehicle 110 by using data received in the computing device 115 (e.g., sensor data from sensors 116, server computer 120, etc.). For example, the computing device 115 can include a program to regulate vehicle 110 operating behaviors (e.g., speed, acceleration, deceleration, steering, etc.) and strategic behaviors, such as the distance between vehicles and / or the amount of time between vehicles, lane changes, minimum gaps between vehicles, minimum paths for left turns, arrival times at specific locations, and minimum arrival times at intersections (without signals) to cross intersections.

[0026] The term controller as used herein includes a computing device that is typically programmed to control a particular vehicle subsystem. Examples include a powertrain controller 112, a brake controller 113, and a steering controller 114. The controller may be, for example, a known electronic control unit (ECU), possibly including additional programming as described herein. The controller may be communicatively connected to a computing device 115 and receive instructions from the computing device to actuate a subsystem in accordance with the instructions. For example, a brake controller 113 may receive instructions from a computing device 115 to operate the brakes of the vehicle 110.

[0027] The one or more controllers 112, 113, 114 of the vehicle 110 may include known electronic control units (ECUs) or the like, including (as non-limiting examples) one or more powertrain controllers 112, one or more brake controllers 113, and one or more steering controllers 114. Each of the controllers 112, 113, 114 may include a respective processor and memory and one or more actuators. The controllers 112, 113, 114 may be programmed and connected to a vehicle 110 communication bus, such as a controller area network (CAN) bus or a local interconnect network (LIN) bus, to receive instructions from the computer 115 and control the actuators based on the instructions.

[0028] Sensors 116 may include a variety of devices known to provide data via a vehicle communication bus. For example, a radar fixed to the front bumper (not shown) of vehicle 110 may provide the distance from vehicle 110 to the next vehicle in front of vehicle 110, or a global positioning system (GPS) sensor disposed in vehicle 110 may provide the geographic coordinates of vehicle 110. Computing device 115 may use the distance provided by radar and / or other sensors 116 and / or the geographic coordinates provided by the GPS sensor to autonomously or semi-autonomously operate vehicle 110.

[0029] The vehicle 110 is generally a land-based autonomous vehicle 110 having three or more wheels, such as a passenger car, a light truck, etc. The vehicle 110 includes one or more sensors 116 , a V to I interface 111 , a computing device 115 , and one or more controllers 112 , 113 , 114 .

[0030] The sensors 116 may be programmed to collect data related to the vehicle 110 and the environment in which the vehicle 110 operates. By way of example and not limitation, the sensors 116 may include, for example, altimeters, cameras, lidars, radars, ultrasonic sensors, infrared sensors, pressure sensors, accelerometers, gyroscopes, temperature sensors, pressure sensors, Hall sensors, optical sensors, voltage sensors, current sensors, mechanical sensors (e.g., switches), etc. The sensors 116 may be used to sense the environment in which the vehicle 110 operates, such as weather conditions, road grade, location of the road, or location of neighboring vehicles 110. The sensors 116 may further be used to collect data, including dynamic vehicle 110 data related to the operation of the vehicle 110, such as speed, yaw rate, steering angle, engine speed, brake pressure, oil pressure, power levels applied to controllers 112, 113, 114 in the vehicle 110, connectivity between components of the vehicle 110 and electrical and logical health.

[0031] Tasks that may be performed by a computing device 115 in a vehicle 110 to enhance the computing device 115's ability to drive the vehicle 110 and assist an occupant in driving the vehicle 110 may include detecting and recognizing traffic signs. A traffic sign may be a sign that conveys information, instructions, or warnings to an occupant while the occupant is driving the vehicle. Traffic signs may include, for example, "Speed ​​Limit" signs, "No Entry" signs, "One Way" signs, and "Stop" signs, and may provide information that permits safe and legal driving of the vehicle 110. Traffic signs may be positioned to be visible to an occupant driving the vehicle 110 and may be configured with colors and shapes that are easily recognizable to the occupant, such as a stop sign. Detecting and recognizing traffic signs may include determining the location of the traffic sign relative to the vehicle 110 and determining the identity and semantics associated with the traffic sign. For example, detecting and recognizing a "Speed ​​Limit" sign may include determining that the sign is adjacent to a road and therefore applies to a vehicle 110 driving on the road, and determining that the sign includes the phrase "Speed ​​Limit 50MPH", which establishes a maximum legal speed of 50 miles per hour for the vehicle 110 on the road.

[0032] Figure 2 2 is a black and white rendering of a color image 200 of a scene viewed from vehicle 110, the color image including a traffic sign 202, in this example a "No Parking" sign. Color image 200 may be acquired by computing device 115 in vehicle 110 using a color camera, which may be, for example, a color video camera. For example, color image 200 may be a single frame of color video data or a composite of more than one frame of color video data to improve the signal-to-noise ratio. Color image 200 may be processed by a computing device according to the techniques disclosed herein to determine that traffic sign 202 is present in color image 200. Detecting and recognizing traffic sign 202 may assist computing device 115 in driving vehicle 110 in a safe and legal manner.

[0033] The detection and recognition of traffic signs is facilitated by the colors and color arrangements used in the traffic signs. For example, an octagonal shape having a predominantly red color and having white lettering is typically reserved for a "Stop" sign, while a rectangular shape having a predominantly white color and having bold lettering may be an informational sign such as a "Speed ​​Limit" sign. Other signs, such as traffic sign 202, may combine multiple colors, in this example white, black, and red in a particular shape, to inform occupants that parking is not permitted near traffic sign 202. The techniques disclosed herein advantageously accurately determine the colors included in color image 200 to detect and recognize traffic signs, such as traffic sign 202.

[0034] Figure 3302 is a diagram of a black and white rendering of a color image 302, which shows how the color image 302 is represented in the computing device 115 memory by three RGB images 304, 306, 308 representing image data of the red, green and blue (RGB) color components of each pixel in the color image 302. The use of three RGB images 304, 306, 308 corresponding to the RGB color components to represent the color image 302 is a technique for representing the entire range of colors perceived by humans. Each color perceived by humans can be approximated as a pigment or light by a combination of RGB components, depending on whether additive or subtractive color is used. Combining RGB components can produce all colors that can be perceived by humans within the limits of the resolution of the RGB components.

[0035] Representing the color image 302 using three RGB images 304, 306, 308 is one of many representations that can accurately represent the color image 302. Other color representations encode the color image 302 by representing the hue, saturation, and brightness (HIS) of each color pixel in the color image 302 (for example) to produce a different representation of the color image 302 that can also accurately represent the colors perceived by humans. In other examples disclosed herein, the color image 302 encoded as three RGB images 304, 306, 308 can be re-encoded to represent the color image 302 using 11 image planes, where each of the 11 image planes represents a single color.

[0036] Figure 4 4 is a diagram of a technique for converting a three-element RGB vector representing the color of a pixel in a color image 302 into an 11-element vector by color space conversion 400. The RGB vector can be converted into a color name vector having fewer or more than 11 elements. Selecting an 11-dimensional color name space having an 11-element color name vector can fully span the human visual perception space using the minimum number of elements that provide good specificity of the color, that is, the color represented by the RGB vector can be converted into an 11-element vector such that most of the sum of the 11 elements is concentrated in one element or two elements. The color space conversion 400 converts the three-element RGB vector 402 representing the color pixel into an 11-element color name vector 404, where each element of the 11-element color name vector 404 is the probability that the value in the three-element RGB vector 402 represents each color of the 11-element color name vector.

[0037] like Figure 4As shown in , each element of the 11-element color name vector represents a common color, namely, black, blue, brown, gray, green, orange, pink, purple, red, white, and yellow. The advantage of transforming the three-element RGB vector 402 into the 11-element color name vector 404 is that the color can be represented by a vector having a large value in one element of a single pixel and having near-zero values ​​in the other 10 elements. After converting the original RGB image to a color name image, a color component map (CCM) can be generated by calculating the similarity between the color name image and the color components, which is Figure 8 Shown in. Figure 8 8 is an exemplary process 800 for generating a color component map (CCM) 826 from an input color name vector map 802. For example, the three color components of a no parking sign may be red, black, and white. Therefore, if the target traffic sign is a no parking sign, three CCMs 826 will be generated for each input image 802. Each CCM indicates a valid color (red, black, or white).

[0038] Return to Figure 4 , the color space conversion 400 transform can be pre-trained using color images of natural color scenes to form a transformation matrix. The RGB values ​​assigned to the colors in the scene of the color image 302 acquired by the color camera may vary due to changes in the color camera and changes in the acquisition electronics, which may cause the color camera to acquire different RGB values ​​from the same scene. In other instances, differences in lighting may cause different RGB values ​​to be acquired in response to the same color. To compensate for these differences, a transformation matrix is ​​obtained, in particular, by training using natural color images in an offline manner, and the 3-element RGB vector 402 is transformed into an 11-element color name vector 404 using the transformation matrix. K-means clustering can be used to train the traffic sign and each color present in the traffic sign to determine clusters of groups of pixels representing the same color, which are color components of the traffic sign.

[0039] K-means clustering is an unsupervised learning technique that groups data based on segmenting the data into discrete clusters by determining K centroids (also denoted as color basis) in color space, and then determining the probability that a particular color (e.g., K=3 for a no parking sign) represented by an 11-element color name vector 404 belongs to a particular color group by measuring the distance of the 11-element color name vector 404 from the 11-dimensional centroids of the clusters and assigning the elements to the closest cluster accordingly. The clusters (colors) may be determined iteratively to maximize the distance between clusters and minimize the number of incorrectly assigned elements. Fig. 9A block diagram is shown in . K-means clustering using traffic sign images can further reduce the variation of acquisition electronics because the color components are trained using cropped traffic sign images. By inputting training data from natural images with different variations, transformation parameters for producing an 11-element color name vector 404 from a 3-element RGB vector 402 can be determined. The 3-element RGB vector 402 can be transformed into an 11-element color name vector 404 using the determined transformation, thereby transforming a color image represented by three (RGB) image planes into 11 color name planes. Color components that indicate the most important colors of a target traffic sign can be determined by K-means clustering. A color component map (CCM) can be determined by calculating the similarity between the color name image and the color components, which is shown in Figure 8 Each single value in the CCM shows the similarity between the color component and the color name image; each CCM has the same X, Y size as the input color image 200.

[0040] Each CCM represents the similarity to one color component of the traffic sign. The advantage of this representation is that, for the colors of interest in traffic signs, each color can be represented by a CCM that has large values ​​in one plane and is otherwise zero, making it very effective to recognize traffic signs by color. Figure 5 is generated using the CCM generation module described above from Figure 2 FIG. 2 is a diagram of three color component maps obtained from a color image 200 . Figure 5 The three color component maps 502, 504, 506 in the figure represent a "red" color component map 502, a "black" color component map 504, and a "white" color component map 506, all of which are obtained from the color image 200. A "1" pixel of the "red" color component map 502, the "black" color component map 504, and the "white" color component map 506 represents the probability that the pixel at that position in the color image 200 is red, black, or white, respectively. The color component maps 502, 504, 506 are formed by the following operations: first, the RGB pixels in the color image 200 are transformed into a color name image as described above; the root mean square error is calculated relative to the color basis; and then the values ​​are multiplied by -1, added to 1, and then calculated to the power of e or the natural logarithm so that higher values ​​represent shorter distances in the 11-dimensional color space, as with respect to Figure 8 shown.

[0041] Figure 6is a diagram illustrating how spatial correspondence between color component maps can be used to enhance the identification of colors in an acquired color image. First, the color component maps 602, 604, 606 are thresholded using Otsu thresholding to suppress low-value color probabilities, thereby forming a binary result. Thresholding refers to converting the value of each pixel of the input image (in this case, the color component maps 602, 604, 606) (which can range from 0 to a maximum value) to a binary value that can be 0 or 1, depending on whether the pixel value is greater than or less than a predetermined threshold. Otsu thresholding assumes that the color probabilities in each color component map 602, 604, 606 are obtained from a binary distribution, and forms a binary image in which each pixel is converted to 1 or 0, depending on which binary distribution the binary image is determined to be obtained from. The binary outputs are then combined algebraically to confirm that all colors appearing in the traffic sign 202 are found in the same region by applying a moving window to the binary color component map to form a binary image such as Figure 6 Input windows 602, 604, 606 shown in FIG.

[0042] Figure 6 The three input windows 602, 604, 606 of FIG. 6 surround corresponding points of three binary color component maps corresponding to the "red" color, the "black" color, and the "white" color, respectively. In order to appear as a "1" in the output windows 610, 612, 614, a pixel must not only be a "1" in the input windows 602, 604, 606, but must also have at least one "1" that appears somewhere in the two other input windows 602, 604, 606 corresponding to two other colors. This means that each "1" pixel in the output windows 610, 612, 614 may represent a traffic sign because all the colors included in a given traffic sign 202 are included in the area near the pixel. This operation is represented by arrow 608, which shows that each output window 610, 612, 614 is an algebraic combination of the three input windows 602, 604, 606. Additionally, the three input windows 602, 604, 606 are segmented into quadrants 616, 618, 620, respectively, and each quadrant is required to have at least one non-zero pixel in order for the pixel to be output as a "1" in one of the output windows 610, 612, 620. After this step, the three output windows 610, 612, 614 are combined 618 by taking a logical "OR" of the three output windows 610, 612, 614 to form one output image 616 representing pixels belonging to the traffic sign 202.

[0043] Figure 7is a diagram of a temporal consistency process applied to two or more output images 710, 722, the two or more output images being formed from input color images 702, 704 acquired at different times while driving the vehicle 110. Optical flow processing is first applied to the input color images 702, 704. The optical flow processing may determine where objects in the scene have translated within the input color images 702, 704 based on information related to the position of the camera and the direction the camera was pointing when the color images 702, 704 were acquired, as well as parameters of the camera lens (including magnification). The optical flow processing relies on, for example, detecting features based on objects in the acquired color images 702 using machine vision techniques, for example, the machine vision techniques determining the same feature in the first acquired color image 702 and the second acquired color image 704. The optical flow processing may determine how features and objects from which the features are derived have translated within the color images 702, 704 based on determining the movement of the camera in space during the time between the acquisition of the color images 702, 704.

[0044] The optical flow processing applied to the input color images 702, 704 is illustrated by an optical flow image 706, which includes optical flow arrows 708 representing the movement of features between the two input color images 702, 704. The optical flow arrows 708 may be determined by the computing device 115 based on sensors 116 (including, for example, accelerometers and GPS) that detect the motion of the vehicle 110 in combination with information about the magnification of the camera lens. By determining the camera movement in space between acquiring the two color images 702, 704 and knowing the magnification of the lens, the change in the position of the features in the two color images 702, 704 may be predicted. The change in the position of the detected features in the color images 702, 704 is shown by the optical flow arrows 708.

[0045] The temporal consistency processing performs optical flow processing on two or more input color images 702, 704 acquired at different times to form an optical flow image 706 including an optical flow arrow 708. In addition, the optical flow image 706 may be obtained by Figure 6The input color images 702, 704 are processed according to the discussed spatial conformance process to produce output images 710, 722. Optical flow arrows 708 representing the motion of features between the input color images 702, 704 can be applied to the output image 710 to translate the detected color objects 712, 714, thereby forming an output image 716 including the color object corresponding to the traffic sign 718 and extraneous color noise 720. Translating the color objects 712, 714 according to the optical flow arrows 708 from the optical flow image 706 can separate the color objects 712, 714 corresponding to real objects in space (such as the traffic sign 718) from the extraneous noise 728, 726 because the real objects in space will translate according to the determined optical flow, while the extraneous noise will not translate.

[0046] The output image 722 includes a color object corresponding to the traffic sign 724, as well as extraneous noise 728, 726. The output image 730 is formed by taking the logical AND of the output image 722 and the translated output image 716. Because the traffic signs 718, 724 appear in both output images 716, 722 and have been translated according to the optical flow arrows 708, the traffic signs 718, 724 will appear at the same location in each image, and will therefore appear in the output image 730 as the traffic sign 732. The extraneous noise 720, 726, 728 will not appear at the same location in the images in both output images 716, 722, and will therefore not appear in the output image 730. In this way, temporal consistency processing can predict where pixels representing traffic signs 718, 724 may appear in output images 716, 722 based on the optical flow processing applied to the acquired color images 702, 720. Because random, extraneous noise pixels do not move according to the optical flow processing, they can be filtered out by this process, while pixels associated with traffic signs 718, 724 are exempt from filtering and thus appear in output image 730 as traffic sign 732 pixels.

[0047] The computing device 115 can determine the position and orientation of a camera included in the vehicle 110 by acquiring information from sensors 116, such as an accelerometer, an electronic compass, and a GPS, and combining this information with information about the magnification of the camera lens to predict where features will move between color images 702, 704 acquired at different times. Predicting the position of one or more features associated with the traffic sign 728 can be used to confirm that the pixels associated with the traffic signs 718, 722 represent the same sign and are therefore temporally consistent. Optical flow processing can determine the temporal consistency of objects detected by spatial coincidence, which means that the detected object represents the same object (traffic sign) in two or more color images 702, 704 based on both spatial coincidence and temporal consistency.

[0048] Figure 8 1 is an exemplary process 800 for generating a color component map (CCM) 826 (CCM generation) from an input color name vector map 802. For example, the three color components of a no parking sign may be red, black, and white. Thus, if the target traffic sign is a no parking sign, three CCMs 826 for each input color name space image 802 will be generated. Each CCM 826 plane indicates a valid color (red, black, or white). In a subtraction stage 808, the input color name vector map 802 is normalized by subtracting a color basis 804, which is formed by process 1000 and processed by color space conversion 806 before being subtracted from the input color name vector map 802.

[0049] At stage 810, for example, the computing device 115 may calculate the root mean square error for each point in the input color name vector map 802 from each of the three color components of the no parking sign and output the results to a distance map 812. The distance map may be made more amenable to computational processing by multiplying each element of the distance map 812 by -1 814 at a multiplication stage 816 and then adding each element of the distance map 812 to 1 before raising each element of the distance map 812 to a power. Following this stage, the exponentiated elements may be normalized again, for example, at a normalization stage 824 by subtracting an average over one or more frames of data. Following the normalization stage 814, the CCM 826 may be output to the computing device 115 for further processing, for example, as described with respect to FIG. Figure 6 Spatial conformity processing described.

[0050] Fig. 9 is a process 900 for detecting a traffic sign Figures 2 to 7The process 900 may be implemented by a processor of the computing device 115, for example, taking information from the sensor 116 as input, and executing instructions and sending control signals via the controllers 112, 113, 114. The process 900 includes a plurality of steps taken in the order disclosed. The process 900 may alternatively include an implementation including fewer steps or may include steps taken in a different order.

[0051] Process 900 begins at step 902, where computing device 115 in vehicle 110 may acquire and store color image 702. Figure 2 As discussed above, for example, the color image 702 may be acquired from a color camera as one or more frames of color video data. Figure 3 and Figure 4 The discussed trained color space conversion 400 converts the pixels of the color image 702 represented by the three-element RGB vector 402 into an 11-element color name vector 404. The 11-element color vector 404 is represented by Figure 8 The framework (CCM generation) shown in FIG. 5 is used to form color coordinate maps (CCMs) 502 , 504 , 506 corresponding to the colors found in traffic signs.

[0052] At step 906, the CCM representing the colors found in the traffic sign is compared with the CCM as described above with respect to Figure 5 and Figure 6 The discussed spatial conformance processing is combined to form and output an image 706 that includes pixels representing the traffic sign 708. The spatial conformance processing requires that the pixels representing the traffic sign 708 include nearby colors that have been previously determined to be included in the traffic sign in order to be determined to be pixels representing the traffic sign 708. This process filters out pixels that may include colors that are included in the traffic sign but are not spatially nearby other colors included in the traffic sign.

[0053] At step 908, computing device 115 may translate output image 706 to form translated output image 714, and combine translated output image 714 with output image 732 formed by processing color image 720 acquired at a different time than color image 702 according to the translation determined by optical flow processing to require temporal consistency. In this way, pixels representing traffic sign 716 are required to appear at similar locations in more than one output image after translation, as described above with respect to Figure 7 Discussed.

[0054] At step 908, pixels representing the detected traffic sign 740 may be output by the computing device 115 for use in driving the vehicle 110 or assisting an occupant in driving the vehicle 110. After this step 908, the process 900 ends.

[0055] Fig.10 is a process for training the color component basis of traffic signs 1000 about Figure 4 and Figure 5 The process 1000 may be implemented by a processor of a computing device 115, for example, taking information from a sensor 116 as input, and executing instructions and sending control signals via controllers 112, 113, 114. The process 1000 includes a plurality of steps taken in the order disclosed. The process 1000 also includes implementations that include fewer steps or may include steps taken in a different order.

[0056] Process 1000 begins at step 1002, where computing device 115 acquires and stores a plurality of color images 200 including traffic signs 202. Process 1000 is an example of unsupervised learning, where there are no labels in the input color images 200.

[0057] At step 1004 , the human operator selects pixels representing the traffic sign 202 and sets all other pixels of the image to zero. In other examples, the computing device 115 may use machine vision techniques to select the traffic sign 202 in the color image 200 .

[0058] In step 1006, as described in Figure 4 As discussed, the selected pixel is converted from the three-element RGB vector 402 to the 11-element color vector 404. In step 1008, the 11-element color vector 404 is grouped into clusters of pixels of similar colors using K-means clustering as described above. Grouping pixels of similar colors can generate parameters for modifying the color space conversion transformation, thereby reliably and accurately grouping pixels associated with traffic signs into the same group. In this way, color bases with variations can be trained into the same group, thereby making the detection of traffic signs acquired under different conditions more accurate and repeatable. The computing device 115 can store the modified color space conversion parameters for use in processing the acquired color image. After this step 1008, the process 1000 ends.

[0059] Fig.11 is a process 1100 for performing spatial conformance processing on the acquired color image 200 Figures 1 to 7The process 1100 may be implemented by a processor of a computing device 115, for example, taking information from a sensor 116 as input, and executing instructions and sending control signals via controllers 112, 113, 114. The process 1100 includes a plurality of steps taken in the order disclosed. The process 1100 also includes implementations that include fewer steps or may include steps taken in a different order.

[0060] Process 1100 begins at step 1102, where the Figure 2 As described above, the computing device 115 may acquire the color image 200. The color image 200 is converted into a color component map representing the colors included in the color image 200 by performing color space conversion using a trained color space conversion transform and then calculating the similarity with a predetermined traffic sign, as described above with respect to Fig. 9 Discussed.

[0061] At step 1104, a CCM representing the traffic sign to be detected is calculated. For example, if a "No Parking" traffic sign is to be detected, then CCMs representing the colors "red", "black", and "white" may be generated.

[0062] In step 1106, as described in Figure 6 As discussed, a CCM representing a desired color may be enhanced, ie, multiplied by -1, added by 1, and the antilog of the resulting value is taken to adjust the values ​​in the CCM for further processing.

[0063] At step 1108, the enhanced CCM may be thresholded as described above with respect to Figure 6 As discussed above, a binary image is formed, wherein each value in the image is either "0" or "1". After this step, in step 1110, as described with respect to Figure 6 As discussed, the binary CCMs representing colors may be combined to retain only those pixels as “1” that appear spatially consistent with other colors from the traffic sign. After this step, process 1100 ends.

[0064] Fig.12 is a process 1200 for performing time consistency processing Figures 1 to 7 The process 1200 may be implemented by a processor of the computing device 115, for example, taking information from the sensor 116 as input, and executing instructions and sending control signals via the controllers 112, 113, 114. The process 1200 includes a plurality of steps taken in the order disclosed. The process 1200 also includes implementations that include fewer steps or may include steps taken in a different order.

[0065] Process 1200 begins at step 1202 , where color image 702 is acquired and stored by computing device 115 .

[0066] In step 1204, the above-mentioned Figure 4 , Figure 5 , Figure 6 , Fig. 9 and Fig.10 The spatial correspondence process described is used to process the color image 702 to produce a binary output image 706 .

[0067] At step 1206 , the computing device 115 may acquire and store the second color image 720 , and at step 1208 , the second color image 720 may be processed using a spatial correspondence process to produce a second binary output image 724 .

[0068] In step 1210, as described above with respect to Figure 7 As discussed, the computing device may determine optical flow parameters of features associated with traffic signs in the input color images 702, 720 to determine translation parameters to be applied to the binary output images 706, 720. Either of the binary output images 706, 720 may be translated to match the other as long as the pixels representing the traffic signs 716, 732 are aligned so that the pixels representing the traffic signs 716, 732 demonstrate temporal consistency when the motion of the camera is compensated for by the determined optical flow parameters.

[0069] Temporal consistency is checked at step 1212, where a logical AND of the binary output images 706, 720 is formed to determine whether the positions of the pixels representing the traffic signs 716, 732 appear at the same location. If the answer is "yes," then the process 1200 branches to step 1214, where the detection of the traffic sign 202 is confirmed by a non-zero output from the logical AND between the binary output images 706, 720. In the event that a non-zero pixel is not output from the logical AND process, the "no" branch is taken and the process 1200 ends.

[0070] Computing devices (such as the computing devices discussed herein) generally each include instructions that can be executed by one or more computing devices (such as the computing devices identified above) and used to implement the blocks or steps of the processes described above. For example, the process blocks discussed above can be embodied as computer-executable instructions.

[0071] Computer-executable instructions may be compiled or interpreted from a computer program created using a variety of programming languages ​​and / or technologies including, without limitation, Java, TM, C, C++, Python, Visual Basic, Java Script, Perl, HTML, etc. In general, a processor (e.g., a microprocessor) receives instructions, such as from a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored in files and transmitted using a variety of computer-readable media. Files in a computing device are generally a collection of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0072] Computer-readable media include any medium that participates in providing data (e.g., instructions) that can be read by a computer. Such media can take many forms, including (but not limited to) non-volatile media, volatile media, etc. Non-volatile media include (for example) optical or magnetic disks and other persistent memories. Volatile media include dynamic random access memory (DRAM), which usually constitutes main memory. Common forms of computer-readable media include (for example) floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAM, PROMs, EPROMs, FLASH-EEPROMs, any other memory chips or boxes, or any other media from which a computer can read.

[0073] All terms used in the claims are intended to have their plain and ordinary meanings as understood by those skilled in the art, unless expressly indicated to the contrary herein. Specifically, use of singular articles such as "a", "the", "said", etc. should be interpreted as reciting one or more of the indicated elements, unless a claim recites an explicit limitation to the contrary.

[0074] The term "exemplary" is used herein in the sense of representing an example; for example, reference to an "exemplary widget" should be interpreted as simply referring to an example of a widget.

[0075] The adverb "about" modifying a value or result means that the shape, structure, measurement results, value, determination, calculation, etc. may deviate from the exactly described geometry, distance, measurement results, value, determination, calculation, etc. due to imperfections in materials, machining, manufacturing, sensor measurement, calculation, processing time, communication time, etc.

[0076] In the drawings, the same reference numerals indicate the same elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, etc. described herein, it should be understood that, although the steps of the processes, etc. have been described as occurring according to a sequence of a specific order, the processes may be practiced using the steps described in an order other than the order described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the processes herein is provided to illustrate certain embodiments and should never be construed as limiting the claimed invention.

[0077] According to the present invention, a method is provided: acquiring a color image; transforming the color image into a color component map; and processing the color component map to detect a traffic sign by determining spatial coincidence and temporal consistency between the color component map and the traffic sign.

[0078] According to an embodiment, the above invention is also characterized in that a color image is transformed into a color component map by converting a three-element RGB vector representing a pixel of the color image into an eleven-element color name vector based on RGB to color component transformation.

[0079] According to an embodiment, each eleven-element color name vector represents a single color.

[0080] According to an embodiment, the color basis is generated by K-means clustering using training data that determines the probability of occurrence of an eleven-element color name vector.

[0081] According to an embodiment, the training data comprises images containing traffic signs.

[0082] According to an embodiment, the above invention is also characterized by processing a color component map by segmenting the color component map using thresholding.

[0083] According to an embodiment, the above invention is also characterized by processing a color component map based on spatial correspondence, wherein segmented color components are retained based on matching color components included in a traffic sign while spatially nearby color components match color components included in the traffic sign.

[0084] According to an embodiment, the above invention is also characterized by: acquiring a second color image; transforming the second color image into a second color component map; segmenting the second color component map; determining spatial consistency and determining temporal consistency based on comparing the color component map from the second image with the color component map from the first image.

[0085] According to an embodiment, determining temporal consistency preserves color components present in the traffic sign in two or more color images acquired at different times.

[0086] According to an embodiment, traffic signs are detected based on determining optical flow, where the optical flow predicts where traffic signs appear based on information related to sensor movement and camera lenses.

[0087] According to the present invention, a computer device is provided, which is programmed to: acquire a color image; transform the color image into a color component map; and process the color component map to detect a traffic sign by determining spatial coincidence and temporal consistency between the color component map and the traffic sign.

[0088] According to an embodiment, the above invention is also characterized by being programmed to transform a color image into a color component map by converting a three-element RGB vector representing a pixel of the color image into an eleven-element color name vector based on an RGB to color component transformation.

[0089] According to an embodiment, each eleven-element color name vector represents a single color.

[0090] According to an embodiment, the color basis is generated by K-means clustering using training data that determines the probability of occurrence of an eleven-element color name vector.

[0091] According to an embodiment, the training data comprises images containing traffic signs.

[0092] According to an embodiment, the above invention is also characterized by being programmed to process a color component map by segmenting the color component map using thresholding.

[0093] According to an embodiment, the above invention is also characterized by being programmed to process a color component map based on spatial correspondence, wherein the segmented color components are retained based on matching color components included in a traffic sign, while spatially nearby color components match color components included in the traffic sign.

[0094] According to an embodiment, the above invention is also characterized by being programmed to: acquire a second color image; transform the second color image into a second color component map; segment the second color component map; determine the spatial consistency in the second color component map and determine the temporal consistency based on comparing the second color component map with the color component map.

[0095] According to an embodiment, determining temporal consistency preserves color components present in the traffic sign in two or more color images acquired at different times.

[0096] According to an embodiment, traffic signs are detected based on determining optical flow, where the optical flow predicts where traffic signs appear based on information related to sensor movement and camera lenses.

Claims

1. A method for identifying a traffic sign, comprising: Get color images; converting the color image into a color name image, and generating a color component map by calculating similarities between the color name image and color components; and Processing the color component map to detect the traffic sign by determining spatial consistency and temporal consistency between the color component map and the traffic sign includes processing the color component map based on spatial consistency, wherein the segmented color components are retained based on matching the color components included in the traffic sign, while spatially nearby color components match the color components included in the traffic sign, and further includes: acquiring a second color image; transforming the second color image into a second color component map; segmenting the second color component map; determining spatial consistency based on comparing the color component map from the second color image with the color component map from the color image and determining temporal consistency.

2. The method of claim 1 further comprises transforming the color image into a color component map by converting a three-element RGB vector representing a pixel of the color image into an eleven-element color name vector according to an RGB to color name space transformation determined based on training data.

3. The method of claim 2, wherein each eleven-element color name vector represents a single color.

4. The method of claim 3, wherein K centroids in color space are generated by K-means clustering using training data, the training data determining the probability of occurrence of the eleven-element color name vector. The method of claim 4 , wherein the training data comprises images containing traffic signs.

6. The method of claim 1, further comprising processing the color component map by segmenting the color component map using thresholding. 7 . The method of claim 1 , wherein determining temporal consistency includes retaining color components present in the traffic sign in two or more color images acquired at different times.

8. The method of claim 7, wherein the traffic sign is detected based on determining optical flow, wherein the optical flow predicts where the traffic sign appears based on information related to sensor movement and camera lenses.

9. The method of claim 5, wherein training comprises performing a k-cluster analysis on the color name vectors.

10. The method of claim 6, further comprising thresholding the color component map based on an Otsu thresholding technique.

11. The method of claim 10, wherein the Otsu thresholding technique produces a binary output image.

12. The method of claim 11, wherein the binary output image is combined with the color image to produce an output color image.

13. A system for recognizing traffic signs, the system comprising a computer programmed to perform the method of any one of claims 1 to 12.

Images