Vision system and method for motor vehicles

Abstract

A vision system for a motor vehicle includes: an imaging device (11) adapted to capture an image (30) of the surrounding environment of the motor vehicle, and a data processing unit (14) adapted to perform image processing on the image (30) captured by the imaging device (11). The data processing unit (14) includes: a traffic sign detector (31) adapted to detect traffic signs in the image (30) captured by the imaging device (11) by image processing; a decision part (35); and a traffic sign estimator (36) adapted to estimate validity information (37) of one or more traffic signs in the image (30) captured by the imaging device (11).

Description

Technical Field

[0001] This invention relates to a vision system for motor vehicles, comprising: an imaging device adapted to capture images from the surrounding environment of the motor vehicle, and a data processing unit adapted to perform image processing on the images captured by the imaging device, wherein the data processing unit includes a traffic sign detector and a decision unit, the traffic sign detector being adapted to detect traffic signs in the images captured by the imaging device through image processing. The invention also relates to corresponding vision methods. Background Technology

[0002] Traffic sign recognition is a crucial component of most modern vehicles. These signs are typically detected and classified using machine learning techniques, such as deep neural networks. Performance is generally good under normal circumstances; however, rare misclassifications can have serious consequences. Map data can supplement image-based information; however, this is not always available or its quality may be too low. Therefore, current autonomous and semi-autonomous vehicles may not be safe enough outside of well-mapped areas.

[0003] In addition, reliability factors such as weather (rain, snow, etc.) and time of day (day / night) can be used to determine the reliability of the current classification, see US 8,918,277 B2. However, using reliability factors can only provide a general estimate of the classification performance based on the current image; it does not address the key issue of individual markers of appearance changes.

[0004] Autonomous vehicles are particularly concerned about situations where drivers are not paying attention to their surroundings, and misclassification of signs can lead to potentially dangerous situations. For example, a 30 km / h speed sign obscured by dirt may be misclassified as an 80 km / h speed sign, potentially causing the vehicle to accelerate far beyond the speed limit.

[0005] Additional risks arise from vandalism and criminal activity, such as placing genuine signs in inappropriate locations or modifying existing signs to cause them to be misclassified in dangerous ways. A special case in this regard is the potential for adversarial attacks against machine learning-based classifiers. Summary of the Invention

[0006] The fundamental problem of this invention is to provide a vision system that has reliable traffic sign recognition suitable for autonomous motor vehicles and self-driving cars.

[0007] This invention addresses this problem through the features of the independent claims. According to the invention, the data processing unit includes a traffic sign estimator adapted to estimate validity information of one or more traffic signs in an image captured by the imaging device. Validity information may include, for example, the probability of the presence of one or more specific traffic signs in the image captured by the imaging device. Alternatively or additionally, validity information may include, for example, information on whether a traffic sign in the image captured by the imaging device is valid, which may be represented, for example, by a corresponding sign bit. Traffic signs may be traffic signs on poles, road signs / road markings, or signs on traffic lights, and therefore can be located anywhere (on traffic poles, on roads, on traffic lights) and can be visible from the imaging device and / or an ego vehicle.

[0008] Human drivers are generally less susceptible to altered signs because they can use common sense to verify the sign's plausibility based on its surrounding environment. Therefore, a human driver is unlikely to, for example, misinterpret a dirty 30 km / h sign as an 80 km / h sign in an urban environment. This common sense is based on a combination of surrounding features such as road type, road curvature, the presence of sidewalks, and buildings. This common sense is precisely what this invention attempts to technically replicate for automated vision systems.

[0009] Preferably, the traffic sign estimator estimates traffic sign validity information based on at least one complete image from the imaging device (i.e., holistically). In this preferred embodiment, the traffic sign estimator can be represented as a holistic traffic sign estimator.

[0010] Information provided by the traffic sign estimator can be compared or combined with information provided by the traffic sign detector / classifier. Based on this combined information, the decision-making part of the data processing unit can take appropriate actions, such as combining information from the traffic sign detector and the traffic sign estimator to initiate an appropriate response, accepting the traffic sign in further processing by the data processing unit, ignoring the traffic sign in further processing by the data processing unit, outputting control signals to the signaling equipment to suggest alternative actions to the driver, and / or outputting control signals to the signaling equipment to signal the driver to take over control of the vehicle.

[0011] This invention applies to autonomous driving, wherein the self-vehicle is an autonomous vehicle adapted to be partially or fully autonomous or automated, and the driver's driving actions are partially and / or completely replaced or performed by the self-vehicle.

[0012] In a preferred embodiment, when the decision unit detects a discrepancy between the detected / classified traffic sign and the estimate from the traffic sign estimator, the decision unit determines which of the sign interpretations provided by the traffic sign detector and the traffic sign estimator is appropriate or most appropriate, wherein further processing by the data processing unit is based on the traffic sign deemed appropriate / most appropriate. In a preferred embodiment, among multiple possible speed signs estimated by the traffic sign estimator, the speed sign with the lowest speed is considered appropriate / most appropriate by the data processing unit and is therefore selected as the true speed sign for further processing. In other words, further processing in the data processing unit is preferably based on selecting a detected traffic sign with the lowest possible speed, i.e., the lowest speed exceeding a predefined probability threshold. Preferably, the decision unit issues a control signal to control the motor vehicle to perform an appropriate action consistent with the traffic sign deemed appropriate / most appropriate. For example, the control signal may control the motor vehicle's braking system to brake, and thus decelerate the motor vehicle until the speed of the appropriate / most appropriate speed sign has been reached.

[0013] This invention can be used to estimate the effectiveness information (e.g., probability or historical probability) of any type of traffic sign (e.g., stop sign, yield sign, priority sign, etc.).

[0014] Preferably, the data processing unit includes a road sign estimator adapted to estimate the validity information (e.g., probability) of one or more road signs and compare the road sign validity information with corresponding road sign detections detected and classified by a road sign detector / classifier. Therefore, the present invention can be used to estimate the validity information of road signs (e.g., left turn, right turn, bus lane, speed, etc.) and compare the (overall) validity information with classified road sign detections in a manner similar to traffic signs.

[0015] This invention can be deployed in motor vehicles in which the driver can take over from the autonomous driving system. In such a scenario, if the decision-making unit detects an inconsistency between the classified detected traffic signs and the estimates of the traffic sign estimator, it can issue a control signal to shut down the autonomous driving system and return control to the driver.

[0016] Preferably, the traffic sign estimator is a classifier, and more preferably a trained classifier. Any kind of machine learning-based classifier can be used for the traffic sign estimator, such as any kind of neural network, such as a convolutional neural network or recurrent neural network, a support vector machine, a propulsion classifier, or a bag-of-words classifier.

[0017] According to one aspect of the invention, the traffic sign estimator is trained on training images that do not include information about the traffic signs of interest (i.e., currently valid traffic signs). For example, if an image has already been taken on a road where a speed limit sign indicates a speed of 80 km / h, the training images should not contain that information. Typically, the traffic sign estimator (especially a neural network) is trained to predict, preferably based on the complete image, the nearest traffic sign that has been passed and is therefore valid for the corresponding image.

[0018] In the case of speed signs or more generally, specific types of traffic signs, this approach is appropriate: start selecting an image from a point where the sign has already been passed by the vehicle and is no longer visible to the imaging system, and continue until the next speed sign or traffic sign of the same specific type just comes into view. For such training images, a traffic sign estimator (particularly a neural network) is trained to predict, preferably based on the complete image, the nearest speed sign or traffic sign of the specific type that has been passed.

[0019] An alternative to training a (holistic) traffic sign estimator is to use an image in which the traffic signs of interest are still visible, but all signs are masked, blurred, or replaced with random signs, in order to avoid making the (holistic) classifier learn to detect actual signs, and instead force it to classify the actual surrounding environment. Attached Figure Description

[0020] Below, the present invention will be described based on preferred embodiments with reference to the accompanying drawings, wherein:

[0021] Figure 1 A schematic diagram of the vision system is shown; and

[0022] Figure 2 A diagram showing the functional elements in the data processing unit of a vision system is provided. Detailed Implementation

[0023] The vision system 10 is preferably an in-vehicle vision system 10 installed or to be installed in or on a motor vehicle. The vision system 10 includes an imaging device 11 for capturing images of an area around the motor vehicle (e.g., the area in front of the motor vehicle). The imaging device 11 or components thereof may be mounted, for example, behind the vehicle's windshield or windshield, in the vehicle's headlights, and / or in the radiator grille. Preferably, the imaging device 11 includes one or more optical imaging devices 12, particularly cameras, which preferably operate in the visible wavelength range, or in the infrared wavelength range, or in both the visible and infrared wavelength ranges. In some embodiments, the imaging device 11 includes a plurality of imaging devices 12, which particularly form a stereoscopic imaging device 11. In other embodiments, only one imaging device 12 forming a monochrome imaging device 11 may be used. Each imaging device 12 is preferably a fixed-focus camera, wherein the focal length f of the lens objective is constant and cannot be varied.

[0024] Imaging device 11 is coupled to data processing unit 14 (or electronic control unit, ECU), preferably an in-vehicle data processing unit 14. Data processing unit 14 is adapted to process image data received from imaging device 11. Data processing unit 14 is preferably a programmed or programmable digital device and preferably includes a microprocessor portion of a microprocessor, microcontroller, digital signal processor (DSP), and / or system-on-a-chip (SoC) device, and preferably has access to or includes digital data memory 25. Data processing unit 14 may include dedicated hardware devices, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), or portions of FPGAs and / or ASICs and / or GPUs in SoC devices, for performing functions such as controlling image capture by imaging device 11, receiving signals containing image information from imaging device 11, reshaping or distorting left / right image pairs for alignment, and / or creating parallax maps or depth images. Data processing unit 14 may be connected to imaging device 11 via a separate cable or vehicle data bus. In another embodiment, one or more of the ECU and imaging devices 12 can be integrated into a single unit, with a preferred option being a one-piece solution including the ECU and all imaging devices 12. All steps, from imaging and image processing to the possible activation or control of safety devices 18, are performed automatically and continuously in real time during driving.

[0025] In another embodiment, the above-described image processing or a portion thereof is performed in the cloud. Therefore, the data processing unit 14 or a portion thereof can be implemented using cloud processing resources.

[0026] Image and data processing performed in data processing unit 14 advantageously includes identifying and preferably also classifying possible objects (object candidates) in front of the motor vehicle, such as pedestrians, other vehicles, cyclists and / or large animals, tracking the position of the objects or object candidates identified in the captured images over time, and activating or controlling at least one safety device 18 depending on an estimate performed on the tracked object (e.g., depending on the estimated collision probability).

[0027] Safety device 18 may include at least one active safety device and / or at least one passive safety device. Specifically, safety device 18 may include one or more of the following: at least one seatbelt tensioner, at least one passenger airbag, one or more restraint systems (e.g., occupant airbags), hood lifter, electronic stability system, at least one dynamic vehicle control system (e.g., brake control system and / or steering control system), speed control system; display device for displaying information related to a detected object; alarm device adapted to provide an alarm to the driver via appropriate optical, acoustic and / or tactile alarm signals.

[0028] Below, for reference Figure 2 The process of traffic sign verification according to the present invention is explained. All method steps related to functional units 31-38 are executed in real time during driving in the data processing unit 14.

[0029] The image 30 captured by the imaging device 11 of the motor vehicle is forwarded to the traffic sign detectors / classifiers 31 and 33 that are known in themselves, and is also forwarded in parallel to the overall traffic sign estimator 36 of the present invention.

[0030] Traffic sign detector 31 is adapted to detect traffic signs in an input image. Traffic signs 32 detected by traffic sign detector 31 are forwarded to traffic sign classifier 33, which is adapted to classify the detected traffic signs into one or more of a predetermined number of categories. Traffic sign classifier 33 is known in itself and typically performs classification on small image patches tightly surrounding the so-called bounding boxes of the detected traffic signs. Classified traffic signs 34 are forwarded to decision unit 35. Traffic sign detector 31 and / or classifier 33 can perform tracking of detected traffic signs across multiple image frames. Traffic sign detector 31 and traffic sign classifier 33 can be a single unit adapted to simultaneously detect and classify traffic signs.

[0031] The overall traffic sign estimator 36 has been pre-trained and is adapted to output validity information 37 for one or more traffic signs in the input image 30 for each complete image from the imaging device 11, such as the probability 37 of the presence of one or more specific (i.e., predefined) traffic signs in the input image 30. More specifically, the overall traffic sign estimator 36 can estimate and output validity information 37 (e.g., probability or validity / invalidity flag bit value) for each of a plurality of predefined traffic signs present in the input image 30. One or more estimated validity values ​​or probabilities 37 are forwarded to the decision section 35. The decision section 35 compares or combines the validity information 37 provided by the traffic sign estimator 36 with the information 34 provided by the traffic sign detector 31 and / or the traffic sign classifier 33, and initiates an appropriate action.

[0032] Below, a practical example is discussed, where the overall traffic sign estimator 36 is restricted to estimating speed signs, and is therefore an overall speed sign estimator 36. Specifically, the overall speed sign estimator 36 is a trained classifier and can be adapted to classify the complete input image into one or more of, for example, five categories: containing a 30 km / h speed sign, containing a 50 km / h speed sign, containing a 60 km / h speed sign, containing an 80 km / h speed sign, and not containing any of these speed signs. It goes without saying that the number of speed signs can be more than five, and / or the speed signs that can be estimated by the overall speed sign estimator 36 can involve other speed signs besides those mentioned above.

[0033] It can be assumed that the traffic sign detectors / classifiers 31 / 33 detect and identify 80 km / h speed signs in a specific input image 30. The overall speed sign estimator 36 estimates the following probabilities for the input image: 5% for 30 km / h speed signs, 15% for 50 km / h speed signs, 40% for 60 km / h speed signs, 30% for 80% speed signs, and 10% for the absence of any of these speed signs.

[0034] The decision part 35 compares or combines the above probability 37 with the findings of the velocity marker detectors / classifiers 31 and 33, and may initiate one or more of the following actions based on the comparison.

[0035] (i) The decision part 35 can accept the detected traffic sign in further processing, either as an 80 km / h traffic sign classified by the speed sign detectors / classifiers 31, 33, or as a 60 km / h speed sign estimated (with the highest probability) by the overall speed estimator 36.

[0036] (ii) Decision section 35 may ignore detected traffic signs during further processing.

[0037] (iii) The decision unit 35 can output control signals 38 to the signaling equipment 18 (see Figure 1 This allows for suggestions to drivers on alternative actions, such as being aware of speed limits.

[0038] (iv) The decision unit 35 can output control signal 38 to the signaling equipment 18 to signal the driver to take over control of the vehicle.

[0039] (v) The decision section 35, which determines the inconsistency between the speed sign (80 km / h) detected and classified by the speed sign detectors / classifiers 31 and 33 and the speed sign (60 km / h) with the highest probability according to the overall speed sign estimator 36, can determine which of the sign interpretations provided by the traffic sign detectors / classifiers 31 and 33 and the traffic sign estimator 36 is appropriate / most appropriate. In one embodiment, the speed sign (60 km / h) with the highest probability according to the overall speed sign estimator 36 can be considered appropriate / most appropriate. In a preferred embodiment, a speed sign (50 km / h) with the lowest speed and a probability exceeding a predetermined threshold (e.g., 10%) is considered appropriate / most appropriate, without considering a speed sign with a probability too low to be considered a true 30 km / h speed sign. The decision section initiates an appropriate action based on the appropriate / most appropriate speed sign, for example, braking the motor vehicle to decelerate it to the speed of the appropriate / most appropriate speed sign.

[0040] As can be clearly seen from the above, the data processing unit 14 preferably includes two different classifiers: a conventional traffic sign classifier 33, which performs classification only on small image patches around the detected traffic signs; and the overall traffic sign estimator 36 of the present invention, which advantageously performs classification on the complete input image.

Claims

1. A vision system for motor vehicles, comprising: An imaging device configured to capture images from the surrounding environment of the motor vehicle; as well as At least one processor is configured as follows: Detect one or more traffic signs in each of the images captured by the imaging device; Traffic sign classifier information associated with the detected traffic signs is generated based on small image patches around the detected traffic signs in each of the images. The validity information of one or more traffic signs is generated based on one or more complete images captured by the imaging device, wherein the validity information indicates the probability that one or more traffic signs are present in the one or more complete images; as well as The control signal is output based at least in part on the validity information and the traffic sign classifier information.

2. The vision system according to claim 1, wherein, The at least one processor is configured to: The traffic sign classifier information is combined with the validity information to output the control signal.

3. The vision system according to claim 1, wherein, The at least one processor is configured to: The traffic sign classifier information is compared with the validity information to output the control signal; Ignore at least one of the detected traffic signs; as well as Output the control signal to: Suggest alternative actions to the driver; or The driver is signaled to take over control of the vehicle.

4. The vision system according to claim 1, wherein, The traffic sign classifier information and the validity information each include one or more traffic sign interpretations, and wherein the at least one processor is configured to: Compare the traffic sign classifier information with the validity information; Based on the comparison, it is determined whether there is a difference between the traffic sign interpretations in one or more traffic sign interpretations of the traffic sign classifier information and the corresponding traffic sign interpretations in one or more traffic sign interpretations of the validity information; and For each identified difference, and based on the associated traffic sign classifier information and validity information, determine which of the one or more traffic sign interpretations included in the traffic sign classifier information and validity information is appropriate.

5. The vision system according to claim 4, wherein, Each of the one or more traffic sign interpretations in the validity information is associated with a traffic speed sign, and wherein the at least one processor is configured to determine that the traffic sign interpretation associated with the traffic speed sign having the lowest speed among the one or more traffic sign interpretations is appropriate.

6. The vision system according to claim 4, wherein, The at least one processor is configured to cause the motor vehicle to perform an appropriate action that conforms to the interpretation of the traffic sign as deemed appropriate.

7. The vision system according to claim 1, wherein, The one or more traffic signs include road markings, and wherein the at least one processor is configured to compare the validity information associated with the road marking with the corresponding traffic sign classifier information associated with the road marking.

8. The vision system according to claim 1, wherein, The at least one processor is further configured to: based on a determination that there is an inconsistency between the traffic sign classifier information and the validity information, disable the autonomous driving system of the motor vehicle and return control to the driver of the motor vehicle.

9. The vision system according to claim 1, wherein, The at least one processor is configured to apply a trained classifier to the image captured by the imaging device to generate the validity information.

10. A computer-implemented method, comprising: Capture images of the environment surrounding the motor vehicle; Detect one or more traffic signs in each of the images; Traffic sign classifier information associated with the detected traffic signs is generated based on small image patches around the detected traffic signs in each of the images. as well as Validity information of one or more traffic signs is generated based on one or more complete images captured by the imaging device, wherein the validity information indicates the probability that one or more traffic signs are present in the one or more complete images; The control signal is output based at least in part on the validity information and the traffic sign classifier information.

11. The computer-implemented method according to claim 10, wherein, The output of the control signal further includes combining the traffic sign classifier information and the validity information.

12. The computer-implemented method according to claim 10, further comprising: The traffic sign classifier information is compared with the validity information to output the control signal; as well as Ignore at least one of the detected traffic signs; as well as Output the control signal to: Suggest alternative actions to the driver; or The driver is signaled to take over control of the vehicle.

13. The computer-implemented method according to claim 10, wherein, The traffic sign classifier information and the validity information each include one or more traffic sign interpretations, and wherein outputting the control signal further includes: Compare the traffic sign classifier information with the validity information; Based on the comparison, it is determined whether there is a difference between the traffic sign interpretations in one or more traffic sign interpretations of the traffic sign classifier information and the corresponding traffic sign interpretations in one or more traffic sign interpretations of the validity information; and For each identified difference, and based on the associated traffic sign classifier information and validity information, determine which of the one or more traffic sign interpretations included in the traffic sign classifier information and validity information is appropriate.

14. The computer-implemented method according to claim 13, wherein, Each of the one or more traffic sign interpretations in the validity information is associated with a traffic speed sign, and wherein the computer-implemented method further includes determining that the traffic sign interpretation associated with the traffic speed sign having the lowest speed among the one or more traffic sign interpretations is appropriate.

15. The computer-implemented method of claim 13, further comprising causing the motor vehicle to perform an appropriate action that conforms to the interpretation of the traffic sign as deemed appropriate.

16. The computer-implemented method according to claim 10, wherein, The one or more traffic signs include road markings, and wherein the at least one processor is configured to compare the validity information associated with the road marking with the corresponding traffic sign classifier information associated with the road marking.

17. A non-transitory machine-readable medium having stored thereon a plurality of executable instructions, which, when executed by a processor, include instructions for performing the following operations: Capture images of the environment surrounding the motor vehicle; Detect one or more traffic signs in each of the images; Traffic sign classifier information associated with the detected traffic signs is generated based on small image patches around the detected traffic signs in each of the images. as well as Validity information of one or more traffic signs is generated based on one or more complete images captured by the imaging device, wherein the validity information indicates the probability that one or more traffic signs are present in the one or more complete images; The control signal is output based at least in part on the validity information and the traffic sign classifier information.

Images