Control system and image analysis system for vehicles

Abstract

A control system for a vehicle, and an image analysis system. Systems and methods use a camera to provide autonomous navigation features. In one embodiment, a lane termination detection system for a vehicle is provided. One or more processing devices associated with the system receive at least one image via a data interface. The device extracts lane termination information from road signs included in the image data, and determines the distance from the vehicle to one or more lane constraints associated with the current lane based on at least one indicator of the position of the vehicle. The processing device determines whether the current lane in which the vehicle is traveling is about to terminate based on the lane termination information and the vehicle position. Further, if the lane in which the vehicle is traveling is about to terminate, the system can prompt the vehicle to change lanes.

Description

[0001] This application is a divisional application of the invention patent application with the original application number 201580017704.2 (International application number: PCT / US2015 / 013812, application date: January 30, 2015, invention name: Vehicle and lane termination detection system and method).

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Patent Application No. 61 / 933,325, filed January 30, 2014; U.S. Provisional Patent Application No. 61 / 993,111, filed May 14, 2014; U.S. Provisional Patent Application No. 62 / 040,246, filed August 21, 2014; U.S. Provisional Patent Application No. 62 / 060,603, filed October 7, 2014; and U.S. Provisional Patent Application No. 62 / 102,669, filed January 13, 2015. The entire contents of all of the foregoing applications are incorporated herein by reference. Technical Field

[0004] The present disclosure relates generally to autonomous vehicle navigation and, more particularly, to systems and methods for providing autonomous vehicle navigation features using a camera. Background Art

[0005] As technology continues to advance, the goal of fully autonomous vehicles that can navigate on the road is about to be achieved. First, the autonomous vehicle can recognize its environment and navigate without operator input. The autonomous vehicle can also consider various factors and make appropriate decisions based on these factors to safely and accurately reach the intended destination. For example, when the vehicle is usually traveling on the road, it will encounter various objects such as other vehicles and pedestrians. The autonomous drive system can identify these objects in the vehicle environment and take appropriate and timely actions to avoid collisions. In addition, the autonomous drive system can recognize other indicators, such as traffic signals, traffic signs and lane markings, which regulate vehicle movement (for example, when the vehicle must stop and can be driven, the speed that the vehicle cannot exceed, the vehicle must be positioned on the road, etc.). The autonomous drive system may need to determine when the vehicle should change lanes, turn at an intersection, change roads, etc. It is obvious from these examples that many factors may need to be resolved in order to provide an autonomous vehicle that can navigate safely and accurately. Summary of the invention

[0006] Embodiments consistent with the present disclosure provide systems and methods for autonomous vehicle navigation. Disclosed embodiments may use cameras to provide autonomous vehicle navigation features. For example, consistent with disclosed embodiments, the disclosed system may include one, two, or more cameras for monitoring the vehicle environment and generating a navigation response based on analysis of images captured by one or more of the cameras.

[0007] In accordance with the disclosed embodiments, a lane termination detection system for a vehicle is provided. The system may include: at least one image capture device configured to acquire at least one image of an area in front of the vehicle, the area including at least one road sign providing information indicating a lane termination; a data interface; and at least one processing device. The at least one processing device may be configured to: receive the at least one image via the data interface; extract lane termination information from a representation of the at least one road sign included in the at least one image; determine a distance from the vehicle to one or more lane constraints associated with a current lane in which the vehicle is traveling based on at least one indicator of the vehicle's position; determine whether the current lane in which the vehicle is traveling is about to terminate based on the extracted lane termination information and based on the determined distance from the vehicle to the one or more lane constraints; and cause the vehicle to change lanes based on the determination of whether the current lane in which the vehicle is traveling is about to terminate.

[0008] Consistent with the disclosed embodiments, the at least one indicator of the position of the vehicle may include a visual determination of a lane among a plurality of lanes in which the vehicle is traveling. The extracted lane termination information may include identifying which lane among a plurality of lanes is about to terminate. Further, the extracted lane termination information may include a distance to the lane termination. Additionally, the extracted lane termination information may include a curvature of a road before the lane terminates. The at least one indicator of the position of the vehicle may include a visual determination of the distance to the lane termination. Alternatively, the at least one indicator of the position of the vehicle may include a visual determination of the distance to a second vehicle, and the second vehicle may be traveling in front of the vehicle. Causing the vehicle to change lanes may include sending instructions to a steering system of the vehicle. Further, the at least one processing device may also be configured to update a distance from the vehicle to one or more lane constraints associated with a new lane of travel based on the at least one indicator of the position of the vehicle.

[0009] Consistent with another disclosed embodiment, a vehicle may include: a body; at least one image capture device configured to acquire at least one image of an area in front of the vehicle, the area including at least one road sign providing information indicating a lane termination; a data interface; and at least one processing device. The at least one processing device may be configured to: receive the at least one image via the data interface; extract lane termination information from a representation of the at least one road sign included in the at least one image; determine a distance from the vehicle to one or more lane constraints associated with a current lane in which the vehicle is traveling based on at least one indicator of the vehicle's position; determine whether the current lane in which the vehicle is traveling is about to terminate based on the extracted lane information and based on the determined distance from the vehicle to the one or more lane constraints; and cause the vehicle to change lanes based on the determination of whether the current lane in which the vehicle is traveling is about to terminate.

[0010] Consistent with disclosed embodiments, the at least one indicator of the vehicle's position may include a visual determination of a lane among a plurality of lanes in which the vehicle is traveling. The extracted lane termination information may include identifying which lane among a plurality of lanes is about to terminate. Further, the extracted lane termination information may include a distance to a lane termination. Additionally, the extracted lane termination information may include a curvature of a road prior to a lane termination. The at least one indicator of the vehicle's position may include a visual determination of a distance to a lane termination. Further, causing the vehicle to change lanes may include sending a command to a steering system of the vehicle.

[0011] In accordance with another disclosed embodiment, a method for detecting a lane termination for a vehicle is provided. The method may include the following steps: acquiring, via at least one image capture device, at least one image of an area in front of the vehicle, the area including at least one road sign providing information indicating a lane termination; extracting lane termination information from a representation of the at least one road sign included in the at least one image; determining a distance from the vehicle to one or more lane constraints associated with a current lane in which the vehicle is traveling based on at least one indicator of the vehicle's position; determining whether the current lane in which the vehicle is traveling is about to terminate based on the extracted lane termination information and based on the determined distance from the vehicle to the one or more lane constraints; and causing the vehicle to change lanes based on the determination of whether the current lane in which the vehicle is traveling is about to terminate.

[0012] Consistent with disclosed embodiments, the at least one indicator of the position of the vehicle may include a visual determination of a lane among a plurality of lanes in which the vehicle is traveling. The extracted lane termination information may include identifying which lane among a plurality of lanes is about to terminate. Further, the extracted lane termination information may include a distance to a lane termination. The method may also include updating a distance from the vehicle to one or more lane constraints associated with a new lane of travel based on the at least one indicator of the position of the vehicle.

[0013] In accordance with the disclosed embodiments, a driver assistance object detection system for a vehicle is provided. The system may include: at least one image capture device configured to acquire a plurality of images representing an area surrounding the vehicle; a data interface; and at least one processing device. The at least one processing device may be configured to: receive a first image from the plurality of images and at least a second image from the plurality of images via the data interface; analyze the first image and at least the second image to determine a reference plane corresponding to a road plane; locate a target object in the first image and the second image; determine a size difference in at least one dimension of the target object between the first image and the second image; determine at least a height of the target object using the determined size difference in the at least one dimension; and cause a change in at least a directional course of the vehicle if the determined height of the object exceeds a predetermined threshold.

[0014] Consistent with the disclosed embodiments, the at least one processing device may also be configured to determine a road clearance height of the vehicle. If the determined height of the target object is less than the road clearance height of the vehicle, the at least one processing device may also be configured to determine that the vehicle is able to change the directional route to drive past the object, the object being oriented between the wheels of the vehicle, and to cause the steering system of the vehicle to change the directional route of the vehicle so that the vehicle drives past the object, the object being oriented between the wheels of the vehicle. Further, the image capture device may have a field of view between about 35 degrees and about 55 degrees. The determination of the height of the target object may also be based on the detected speed of the vehicle. In addition, the predetermined threshold may be at least 2 cm from the surface of the road. The at least one processor may also be configured to determine that the road plane is an average surface level associated with the road surface appearing in the plurality of images, and the road surface may include gravel. Further, the determined height of the target object is less than 10 cm relative to the road plane.

[0015] In accordance with another disclosed embodiment, a vehicle may include: a body; at least one image capture device configured to acquire a plurality of images representing an area surrounding the vehicle; a data interface; and at least one processing device. The at least one processing device may be configured to: receive, via the data interface, a first image from the plurality of images and at least a second image from the plurality of images; analyze the first image and at least the second image to determine a reference plane corresponding to a road plane; locate a target object in the first image and the second image; determine a size difference in at least one dimension of the target object between the first image and the second image; determine at least a height of the target object using the determined size difference in the at least one dimension; and cause a change in at least a directional course of the vehicle if the determined height of the object exceeds a predetermined threshold.

[0016] Consistent with the disclosed embodiments, the at least one processing device may also be configured to determine a road clearance height of the vehicle. If the determined height of the target object is less than the road clearance height of the vehicle, the at least one processing device may also be configured to determine that the vehicle is able to change the directional route to drive past the object, the object being oriented between the wheels of the vehicle, and to cause the steering system of the vehicle to change the directional route of the vehicle so that the vehicle drives past the object, the object being oriented between the wheels of the vehicle. Further, the image capture device may have a field of view between about 35 degrees and about 55 degrees. The determination of the height of the target object may also be based on the detected speed of the vehicle. In addition, the predetermined threshold may be at least 2 cm from the surface of the road. The at least one processor may also be configured to determine that the road plane is an average surface level associated with the road surface appearing in the plurality of images, and the road surface may include gravel. Further, the determined height of the target object is less than 10 cm relative to the road plane.

[0017] In accordance with another disclosed embodiment, a method for detecting an object on a road is provided. The method may include the following steps: acquiring, via at least one image capture device, a plurality of images representing an area surrounding the vehicle; receiving a first image from the plurality of images and at least a second image from the plurality of images; analyzing the first image and at least the second image to determine a reference plane corresponding to a road plane; determining a size difference in at least one dimension of the target object between the first image and the second image; determining at least a height of the target object using the determined size difference in at least one dimension; and causing a change in at least a directional course of the vehicle if the determined height of the object exceeds a predetermined threshold.

[0018] Consistent with disclosed embodiments, the predetermined threshold may be at least 2 cm from the surface of the road.

[0019] In accordance with the disclosed embodiments, a traffic light detection system for a vehicle is provided. The system may include: at least one image capture device configured to acquire a plurality of images of a traffic light located in an area above the vehicle, outside the line of sight of a typical driver position of the vehicle; a data interface; and at least one processing device. The at least one processing device may be configured to: receive the plurality of images via the data interface; determine a state of the traffic light based on an analysis of at least a first image from among the plurality of images; determine a change in the state of the traffic light based on at least a second image from among the plurality of images; and cause a system response based on the determination of the change in state.

[0020] Consistent with disclosed embodiments, the at least one image capture device may include a sensor and a lens having a focal length shorter than a short side of the sensor. The at least one processing device may also be configured to correct optical distortion in the at least first image. Further, the vertical field of view from the typical driver position in the vehicle may be approximately 65 degrees, and the vertical field of view of the at least one image capture device may be greater than 65 degrees. The at least one image capture device may be configured for positioning within the vehicle. Additionally, the at least one processing device may be further configured to determine the state of the traffic light when the at least one image capture device acquires the plurality of images through a windshield of the vehicle. The at least one image capture device may be configured to acquire an image of a traffic light positioned in a conic region between 150 and 180 degrees relative to the positioning of the at least one image capture device. Further, the at least one image capture device may be configured to acquire an image of a traffic light positioned in a conic region between 170 and 180 degrees relative to the positioning of the at least one image capture device. The system response may include reporting the change in state to a driver of the vehicle. The system response may also include accelerating to a driver-specified cruising speed. The at least one processing device may also be configured to determine the association of the acquired traffic lights in the plurality of images based on at least one indicator of vehicle position. The at least one indicator of vehicle position may include lane markings identified based on the analysis of the plurality of images. Additionally, the at least one indicator of vehicle position may include a vehicle location acquired via GPS, and the at least one processing device may be configured to compare the vehicle location acquired via GPS with map data to determine the association of the traffic lights.

[0021] In accordance with another disclosed embodiment, a vehicle may include: a body; at least one image capture device configured to acquire a plurality of images of a traffic light located in an area above the vehicle outside of a typical driver's line of sight of the vehicle; a data interface; and at least one processing device. The at least one processing device may be configured to: receive the plurality of images via the data interface; determine a state of the traffic light based on an analysis of at least a first image from among the plurality of images; determine a change in the state of the traffic light based on at least a second image from among the plurality of images; and cause a system response based on the determination of the change in state.

[0022] Consistent with the disclosed embodiments, the at least one image capture device may be configured to locate the interior of the vehicle. The at least one processing device may also be configured to determine the state of the traffic light when the at least one image capture device acquires the plurality of images through a windshield of the vehicle. Additionally, the system response may include reporting the change in state to the driver of the vehicle, or alternatively, may include accelerating to a driver-specified cruising speed.

[0023] In accordance with another disclosed embodiment, a method for navigating a vehicle is provided. The method may include the steps of acquiring, via at least one image capture device, a plurality of images of a traffic light located in an area above the vehicle outside of a typical driver position of the vehicle; determining a state of the traffic light based on an analysis of at least a first image from the plurality of images; determining a change in the state of the traffic light based on at least a second image from the plurality of images; and causing a system response based on the determination of the change in state.

[0024] Consistent with the disclosed embodiments, a traffic light detection system for a vehicle is provided. The system may include: at least one image capture device, the at least one image capture device being configured to acquire at least one image of an area in front of the vehicle, the area including a plurality of traffic light poles each including at least one traffic light. The system may also include a data interface and at least one processing device. The at least one processing device may be configured to: receive the at least one acquired image via the data interface; determine an association of each of the plurality of traffic light poles with the vehicle using at least one indicator of a vehicle position as determined from the at least one acquired image; determine a state of a traffic light included in at least one traffic light pole determined to be associated with the vehicle based on the at least one acquired image; and cause a system response based on the determined state.

[0025] Consistent with disclosed embodiments, the at least one indicator of the vehicle's position may include a lane marking identified based on an analysis of the at least one image. Further, determining the state of the traffic light may include any of the following steps: determining the positioning of the traffic light within an associated traffic light pole; determining whether the traffic light is illuminated; determining the color of the traffic light; and determining whether the traffic light includes an arrow. The system response may include any of the following steps: providing a warning to the driver of the vehicle; applying vehicle brakes; and terminating cruise control. Additionally, the association of each of the plurality of traffic light poles with the vehicle may include indicating an orientation of each of the plurality of traffic light poles relative to the vehicle. The at least one traffic light pole determined to be associated with the vehicle may face the front of the vehicle.

[0026] Consistent with another disclosed embodiment, a vehicle may include: a body; at least one image capture device configured to acquire at least one image of an area in front of the vehicle, the area including a plurality of traffic light poles each including at least one traffic light; a data interface; and at least one processing device. The at least one processing device may be configured to: receive the at least one acquired image via the data interface; determine an association of each of the plurality of traffic light poles with the vehicle using at least one indicator of a vehicle position as determined from the at least one acquired image; determine a state of a traffic light included in at least one traffic light pole determined to be associated with the vehicle based on the at least one acquired image; and cause a system response based on the determined state.

[0027] Consistent with disclosed embodiments, the at least one indicator of the vehicle's position may include a lane marking identified based on an analysis of the at least one image. Further, determining the state of the traffic light may include any of the following steps: determining the positioning of the traffic light within an associated traffic light pole; determining whether the traffic light is illuminated; determining the color of the traffic light; and determining whether the traffic light includes an arrow. The system response may include any of the following steps: providing a warning to the driver of the vehicle; applying vehicle brakes; and terminating cruise control. Additionally, the association of each of the plurality of traffic light poles with the vehicle may include indicating an orientation of each of the plurality of traffic light poles relative to the vehicle. The at least one traffic light pole determined to be associated with the vehicle may face the front of the vehicle.

[0028] In accordance with another disclosed embodiment, a method for traffic light detection is provided. The method may include the steps of acquiring, via at least one image capture device, at least one image of an area in front of a vehicle, the area including a plurality of traffic light poles each including at least one traffic light; determining an association of each of the plurality of traffic light poles with the vehicle using at least one indicator of a vehicle position as determined from the at least one acquired image; determining a state of a traffic light included in at least one traffic light pole determined to be associated with the vehicle based on the at least one acquired image; and causing a system response based on the determined state.

[0029] Consistent with disclosed embodiments, the at least one indicator of the vehicle's position may include a lane marking identified based on an analysis of the at least one image. Further, determining the state of the traffic light may include any of the following steps: determining the positioning of the traffic light within an associated traffic light pole; determining whether the traffic light is illuminated; determining the color of the traffic light; and determining whether the traffic light includes an arrow. The system response may include any of the following steps: providing a warning to the driver of the vehicle; applying vehicle brakes; and terminating cruise control. Additionally, the association of each of the plurality of traffic light poles with the vehicle may include indicating an orientation of each of the plurality of traffic light poles relative to the vehicle. The at least one traffic light pole determined to be associated with the vehicle may face the front of the vehicle.

[0030] In accordance with the disclosed embodiments, a traffic light detection system for a vehicle is provided. The system may include: at least one image capture device configured to acquire at least one image of an area in front of the vehicle, the area including a traffic light pole having at least one traffic light; a data interface; and at least one processing device. The at least one processing device may be configured to: receive the at least one acquired image via the data interface; determine whether the vehicle is in a turn lane based on at least one indicator of the vehicle's position; perform image processing on the at least one image to determine whether an arrow is present in the at least one image within a pixel of the image representing the traffic light; determine a state of the traffic light based on the at least one image; and cause a system response based on the determination of the state of the traffic light whether the traffic light includes an arrow and whether the vehicle is in a turn lane.

[0031] Consistent with disclosed embodiments, the at least one indicator of the vehicle's position may include a lane marking identified based on an analysis of the at least one image. Further, the at least one indicator of the vehicle's position may include an arrow associated with the traffic light pole identified based on an analysis of the at least one image. The at least one indicator of the vehicle's position may include a vehicle location obtained via GPS. Additionally, the system response may include any of the following steps: providing a notification to the driver of the vehicle; applying vehicle braking; suspending cruise control; and initiating an automatic turning maneuver. Moreover, determining the state of the traffic light may include any of the following steps: determining whether the traffic light is illuminated; determining the location of the traffic light within the traffic light pole; and determining a color associated with the traffic light. The at least one indicator of the vehicle's position may include a distance from an intersection. The determined state of the traffic light may be that it is a flashing yellow light, and if so, the at least one processing device may also be configured to determine whether one or more vehicles are approaching the intersection from an opposite direction. If no vehicle is approaching the intersection from an opposite direction, the at least one processing device may also be configured to cause a system response based on the determination, wherein the system response may include any of the following steps: providing a notification to the driver of the vehicle; applying vehicle braking; suspending cruise control; and initiating an automatic turning maneuver. Additionally, the at least one processing device may also be configured to determine whether the turning lane in which the vehicle is located is authorized to proceed in the direction indicated by the determined state of the traffic light. The step of determining whether the turning lane in which the vehicle is located is authorized to proceed may include determining whether to turn left or right, or to proceed straight through the intersection based on the vehicle's intended destination.

[0032] Consistent with another disclosed embodiment, a vehicle may include: a body; at least one image capture device configured to acquire at least one image of an area in front of the vehicle, the area including a traffic light pole having at least one traffic light; a data interface; and at least one processing device. The at least one processing device may be configured to: receive the at least one acquired image via the data interface; determine whether the vehicle is in a turn lane based on at least one indicator of vehicle position; perform image processing on the at least one image to determine whether an arrow is present in the at least one image within a pixel of the image representing the traffic light; determine a state of the traffic light based on the at least one image; and cause a system response based on the determination of the state of the traffic light whether the traffic light includes an arrow and whether the vehicle is in a turn lane.

[0033] Consistent with disclosed embodiments, the at least one indicator of the vehicle's position may include lane markings identified based on analysis of the at least one image. The at least one indicator of the vehicle's position may include a vehicle location obtained via GPS. Additionally, the system response may include any of the following steps: providing a notification to the driver of the vehicle; applying vehicle brakes; disabling cruise control; and initiating an automatic turning maneuver. Moreover, determining the state of the traffic light may include any of the following steps: determining whether the traffic light is illuminated; determining the location of the traffic light within the traffic light frame; and determining a color associated with the traffic light. The at least one indicator of the vehicle's position may include a distance from an intersection. The determined state of the traffic light may be that it is a flashing yellow light, and if so, the at least one processing device may also be configured to determine whether one or more vehicles are approaching the intersection from an opposite direction. If no vehicles are approaching the intersection from an opposite direction, the at least one processing device may also be configured to cause a system response based on the determination, wherein the system response may include any of the following steps: providing a notification to the driver of the vehicle; applying vehicle brakes; disabling cruise control; and initiating an automatic turning maneuver. Additionally, the at least one processing device may be further configured to determine whether the turning lane in which the vehicle is located is authorized to proceed in a direction indicated by the determined state of the traffic light.

[0034] In accordance with another disclosed embodiment, a method for traffic light detection is provided. The method may include the following steps: acquiring, via at least one image capture device, at least one image of an area in front of a vehicle, the area including a traffic light pole having at least one traffic light; determining whether the vehicle is in a turn lane based on at least one indicator of a vehicle position; performing image processing on the at least one image to determine whether an arrow is present in the at least one image within a pixel of the image representing the traffic light; determining a state of the traffic light based on the at least one image; and causing a system response based on the state of the traffic light including an arrow and the determination of whether the vehicle is in a turn lane.

[0035] Consistent with disclosed embodiments, the at least one indicator of vehicle position may include a lane marking identified based on analysis of the at least one image. Further, the at least one indicator of vehicle position may include an arrow associated with the traffic light pole identified based on analysis of the at least one image. The at least one indicator of vehicle position may include a vehicle location obtained via GPS. Additionally, the system response may include any of the following steps: providing a notification to the driver of the vehicle; applying vehicle braking; terminating cruise control; and initiating an automatic turning maneuver. Moreover, determining the state of the traffic light may include any of the following steps: determining whether the traffic light is illuminated; determining the location of the traffic light within the traffic light pole; and determining a color associated with the traffic light.

[0036] In accordance with the disclosed embodiments, a traffic light detection system for a vehicle is provided. The system may include: at least one image capture device configured to acquire multiple images of an area in front of the vehicle, the area including a traffic light pole having at least one traffic light; a data interface; and at least one processing device configured to receive the multiple images via the data interface. The at least one processing device may also be configured to: align regions of the multiple images corresponding to the traffic light based on a determined brightness center; expand each pixel located within the aligned region; determine a set of average pixel values ​​including an average pixel value for each set of corresponding expanded pixels within the aligned region; and determine whether the traffic light includes an arrow based on the set of average pixel values.

[0037] Consistent with disclosed embodiments, the expansion of each pixel within the aligned region may include expanding each pixel into a 3x3 matrix of pixels. Alternatively, the expansion of each pixel within the aligned region may include expanding each pixel into a 4x4 matrix of pixels. The step of aligning the regions of the multiple images corresponding to the traffic light may include rotating the regions of the multiple images. In addition, the at least one processing device may be configured to determine the color of the arrow based on the set of average pixel values ​​when the traffic light includes an arrow. Further, the at least one processing device may be configured to cause a system response based on the determination of the color of the arrow. The system response may include any of the following steps: providing a notification to the driver of the vehicle; applying vehicle braking; terminating cruise control; and initiating an automatic turning maneuver.

[0038] In accordance with another disclosed embodiment, a vehicle is provided. The vehicle may include: a body; at least one image capture device configured to acquire multiple images of an area in front of the vehicle, the area including a traffic light pole having at least one traffic light; a data interface; and at least one processing device configured to receive the multiple images via the data interface. The at least one processing device may also be configured to: align regions of the multiple images corresponding to the traffic light based on a determined brightness center; expand each pixel located within the aligned region; determine a set of average pixel values ​​including an average pixel value for each set of corresponding expanded pixels within the aligned region; and determine whether the traffic light includes an arrow based on the set of average pixel values.

[0039] Consistent with the disclosed embodiments, the expansion of each pixel within the aligned region may include expanding each pixel into a 3x3 matrix of pixels. Alternatively, the expansion of each pixel within the aligned region may include expanding each pixel into a 4x4 matrix of pixels. The step of aligning the regions of the multiple images corresponding to the traffic light may include rotating the regions of the multiple images. In addition, the at least one processing device may also be configured to, when the traffic light includes an arrow, determine the color of the arrow based on the set of average pixel values.

[0040] In accordance with another disclosed embodiment, a traffic light detection method is provided. The method may include the following steps: acquiring, via at least one image capture device, a plurality of images of an area in front of a vehicle, the area including a traffic light pole having at least one traffic light; aligning, based on a determined brightness center, areas of the plurality of images corresponding to the traffic light; and expanding each pixel within the aligned area. The method may also include the following steps: determining a set of average pixel values ​​including an average pixel value for each set of corresponding expanded pixels within the aligned area; and determining, based on the set of average pixel values, whether the traffic light includes an arrow.

[0041] Consistent with disclosed embodiments, the expansion of each pixel within the aligned region may include expanding each pixel into a 3x3 matrix of pixels. Alternatively, the expansion of each pixel within the aligned region may include expanding each pixel into a 4x4 matrix of pixels. The step of aligning the regions of the multiple images corresponding to the traffic light may include rotating the regions of the multiple images. In addition, the at least one processing device may be configured to determine the color of the arrow based on the set of average pixel values ​​when the traffic light includes an arrow. Further, the at least one processing device may be configured to cause a system response based on the determination of the color of the arrow. The system response may include any of the following steps: providing a notification to the driver of the vehicle; applying vehicle braking; terminating cruise control; and initiating an automatic turning maneuver.

[0042] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that are executed by at least one processing device and perform any of the methods described herein.

[0043] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings:

[0045] Figure 1 is a schematic representation of an exemplary system consistent with the disclosed embodiments.

[0046] Figure 2A is a diagrammatic side view representation of an exemplary vehicle including systems consistent with the disclosed embodiments.

[0047] Figure 2B It is consistent with the disclosed embodiment Figure 2A A schematic top view representation of the vehicle and systems shown.

[0048] Figure 2C is a diagrammatic top view representation of another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0049] Figure 2D is a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0050] Figure 2E is a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments.

[0051] Figure 2F is a diagrammatic representation of an exemplary vehicle control system consistent with the disclosed embodiments.

[0052] Figure 3A is a diagrammatic representation of the interior of a vehicle consistent with disclosed embodiments, including a rearview mirror and a user interface for a vehicle imaging system.

[0053] Figure 3B is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against a vehicle windshield consistent with disclosed embodiments.

[0054] Figure 3C It is consistent with the disclosed embodiment Figure 3B Illustrations of camera mounts showing different viewing angles.

[0055] Figure 3D is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against a vehicle windshield consistent with disclosed embodiments.

[0056] Figure 4 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations consistent with the disclosed embodiments.

[0057] Figure 5A is a flow chart illustrating an exemplary process for generating one or more navigation responses based on monocular image analysis consistent with disclosed embodiments.

[0058] Figure 5B is a flow chart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a set of images consistent with the disclosed embodiments.

[0059] Figure 5C is a flow chart illustrating an exemplary process for detecting road signs and / or lane geometry information in a set of images consistent with the disclosed embodiments.

[0060] Figure 5D is a flow chart illustrating an exemplary process for detecting traffic lights in a set of images consistent with disclosed embodiments.

[0061] Figure 5E is a flow chart illustrating an exemplary process for generating one or more navigation responses based on a vehicle path consistent with the disclosed embodiments.

[0062] Fig. 5F is a flow chart illustrating an exemplary process for determining whether a leading vehicle has changed lanes consistent with the disclosed embodiments.

[0063] Figure 6is a flow chart illustrating an exemplary process for generating one or more navigation responses based on stereoscopic image analysis consistent with disclosed embodiments.

[0064] Figure 7 is a flow chart illustrating an exemplary process for generating one or more navigation responses based on three sets of image analysis consistent with disclosed embodiments.

[0065] Figure 8 is a diagrammatic top view representation of an exemplary vehicle including a system consistent with disclosed embodiments in which the vehicle's current lane of travel is about to terminate.

[0066] Fig. 9 is a schematic representation of a memory of an exemplary lane end recognition system consistent with the disclosed embodiments.

[0067] Fig. 10A is a diagrammatic representation of an exemplary vehicle including a lane end recognition system encountering the end of a travel lane marked by a road sign consistent with the disclosed embodiments.

[0068] Fig. 10B is a diagrammatic representation of an exemplary vehicle including a lane end recognition system for traveling a measurable distance behind a leading vehicle consistent with the disclosed embodiments.

[0069] Fig.11 is a flow chart illustrating an exemplary process for determining whether a current lane in which a vehicle is traveling is about to terminate, consistent with the disclosed embodiments.

[0070] Fig.12 is a diagrammatic top view representation of an exemplary vehicle including a system consistent with disclosed embodiments that detects an object within a lane of travel in which the vehicle is traveling.

[0071] Fig.13 is a schematic representation of a memory of an exemplary driver assistance object detection system consistent with the disclosed embodiments.

[0072] Fig.14A is a schematic representation of two acquired images indicating objects found inside a driving lane among a plurality of images acquired by an exemplary driver assistance object detection system consistent with the disclosed embodiments.

[0073] Fig. 14B is a diagrammatic top view representation of an exemplary vehicle including a system consistent with disclosed embodiments that detects an object within a lane of travel in which the vehicle is traveling.

[0074] Fig.15is a flow chart illustrating an exemplary process for determining whether a current lane in which a vehicle is traveling is about to terminate, consistent with the disclosed embodiments.

[0075] Fig.16 is a diagrammatic side view representation of an exemplary vehicle including an image capture device with an extended viewing angle consistent with disclosed embodiments.

[0076] Fig.17 is a diagrammatic top view representation of a vehicle including an image capture device with an extended viewing angle consistent with disclosed embodiments.

[0077] Fig.18 is a schematic representation of a memory storing instructions for identifying objects located outside of a typical driver's line of sight, consistent with disclosed embodiments.

[0078] Fig.19 is a flow chart illustrating an exemplary process for identifying objects positioned outside of a typical driver's line of sight, consistent with the disclosed embodiments.

[0079] Fig. 20 is a diagrammatic representation of two exemplary vehicles approaching an intersection having multiple traffic lights consistent with disclosed embodiments.

[0080] Fig.21 is a diagrammatic representation of an exemplary vehicle approaching an intersection having multiple traffic lights consistent with disclosed embodiments.

[0081] Fig. 22 is a schematic representation of a memory storing instructions for detecting traffic lights consistent with the disclosed embodiments.

[0082] Fig.23 is a flow chart illustrating an exemplary process for detecting traffic lights consistent with the disclosed embodiments.

[0083] Fig.24 is a diagrammatic top view representation of an exemplary vehicle including a system consistent with disclosed embodiments as the vehicle approaches an intersection with a traffic light.

[0084] Fig.25 is a schematic representation of a memory of an exemplary traffic light detection system consistent with the disclosed embodiments.

[0085] Fig.26A is a diagrammatic representation of an exemplary vehicle including a traffic light detection system encountering an intersection having a traffic light and determining the state thereof, consistent with disclosed embodiments.

[0086] Fig.26Bis a diagrammatic representation of an exemplary vehicle including a traffic light detection system encountering an intersection with a flashing yellow light and determining a passing action consistent with disclosed embodiments.

[0087] Fig. 27 is a flow chart illustrating an exemplary process for determining the state of a traffic light at an intersection consistent with disclosed embodiments.

[0088] Fig.28 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations consistent with the disclosed embodiments.

[0089] Fig.29A is an illustration of an exemplary traffic light consistent with disclosed embodiments.

[0090] Fig.29B is another illustration of an exemplary traffic light consistent with disclosed embodiments.

[0091] Fig.29C It is consistent with the disclosed embodiment Fig.29A Illustration of an exemplary image of a portion of a traffic light.

[0092] Fig.29D It is consistent with the disclosed embodiment Fig.29B Illustration of an exemplary image of a portion of a traffic light.

[0093] Fig. 30A is a flow chart illustrating an exemplary process for determining whether a traffic light includes an arrow, consistent with disclosed embodiments.

[0094] Fig. 30B is a flow chart illustrating an exemplary process for inducing a system response based on detection of an arrow signal in a traffic light, consistent with the disclosed embodiments. DETAILED DESCRIPTION

[0095] The following detailed description refers to the accompanying drawings. Where possible, the same reference numerals are used in the drawings and in the following description to refer to the same or similar parts. Although several illustrative embodiments are described herein, modifications, rewrites and other implementations may be made. For example, the components illustrated in the figures may be replaced, added or modified, and the illustrative methods described herein may be modified by replacing, reordering, removing or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the appropriate scope is defined by the appended claims.

[0096] Disclosed embodiments provide systems and methods for providing autonomous navigation features using cameras. In various embodiments, the system may include one, two, or more cameras that monitor the environment of the vehicle. In one embodiment, the system may identify when a driving lane will terminate, and when termination is detected, make responsive navigation changes such as lane changes. The system may operate based solely on visual information, rather than relying on map data to make this determination. Further, while the system may identify the end of a lane by direct observation of the lane termination, the system may also identify the approaching end of a lane earlier by, for example, observing and identifying road sign alerts that provide information about the lane termination. In another embodiment, the system may determine that there are obstacles on the road, wherein the obstacles have a height of less than, for example, 10 cm. In another embodiment, the system may see and identify traffic lights located above the vehicle. In another embodiment, the system may identify a traffic light and its state from among a plurality of traffic lights, and cause a navigation change based on the state of the traffic light (e.g., suspending cruising and causing braking when a red light is identified). For example, the system may identify which imaged traffic lights are associated with the vehicle, and respond only to those traffic lights that are determined to be associated. In another embodiment, the system can identify turning lane traffic lights and their status based on road environment information combined with the determined characteristics of the lights. In another embodiment, the system can use super-resolution technology to identify and analyze traffic lights. For example, the system can distinguish between circular traffic lights and arrow-shaped traffic lights.

[0097] Figure 1 1 is a block diagram representation of a system 100 consistent with an exemplary disclosed embodiment. Depending on the requirements of a particular implementation, the system 100 may include various components. In some embodiments, the system 100 may include a processing unit 110, an image acquisition unit 120, a position sensor 130, one or more memory units 140, 150, a map database 160, and a user interface 170. The processing unit 110 may include one or more processing devices. In some embodiments, the processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, depending on the requirements of a particular application, the image acquisition unit 120 may include any number of image acquisition devices and components. In some embodiments, the image acquisition unit 120 may include one or more image capture devices (e.g., cameras), such as image capture device 122, image capture device 124, and image capture device 126. The system 100 may also include a data interface 128 that communicatively connects the processing device 110 and the image acquisition device 120. For example, the data interface 128 may include any one or more wired and / or wireless links for transmitting image data acquired by the image acquisition device 120 to the processing unit 110 .

[0098] Both the application processor 180 and the image processor 190 may include various types of processing devices. For example, either or both of the application processor 180 and the image processor 190 may include a microprocessor, a preprocessor (such as an image preprocessor), a graphics processor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, a memory, or any other type of device suitable for running applications and image processing and analysis. In some embodiments, the application processor 180 and / or the image processor 190 may include any type of single-core or multi-core processor, a mobile device microcontroller, a central processing unit, etc. Various processing devices may be used, including, for example, those commercially available from, for example, etc., and may include various architectures (e.g., x86 processors, etc).

[0099] In some embodiments, the application processor 180 and / or the image processor 190 may include any commercially available The EyeQ series processor chips. These processor designs each include multiple processing units with local memory and instruction sets. Such a processor may include a video input for receiving image data from multiple image sensors, and may also include video output capabilities. In one example, Using 90nm micron technology operating at 332Mhz. The architecture consists of two floating point, hyperthreaded 32-bit RISC CPUs ( cores), five visual computing engines (VCEs), and three vector microcode processors Denali 64-bit mobile DDR controller, 128-bit internal supersonic interconnect, dual 16-bit video input controller and 18-bit video output controller, 16-channel DMA and several peripherals. MIPS34K CPU manages five VCEs, three VMPs TM and DMA, a second MIPS34K CPU and multi-channel DMA, and other peripherals. Five VCEs, three and MIPS34K CPUs can perform the intensive visual calculations required by multi-function bundled applications. Six times more powerful Can be used in the disclosed embodiments.

[0100] Any of the processing devices disclosed herein may be configured to perform certain functions. The step of configuring a processing device, such as any of the described EyeQ processors or other controllers or microprocessors, to perform certain functions may include programming of computer executable instructions and making these instructions available to the processing device for execution during operation of the processing device. In some embodiments, the step of configuring the processing device may include programming the processing device directly with architectural instructions. In other embodiments, the step of configuring the processing device may include storing executable instructions on a memory accessible to the processing device during operation. For example, the processing device may access the memory during operation to obtain and execute the stored instructions.

[0101] Although Figure 1 Two separate processing devices are depicted as included in processing unit 110, but more or less processing devices may be used. For example, in some embodiments, a single processing device may be used to complete the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by more than two processing devices.

[0102] The processing unit 110 may include various types of devices. For example, the processing unit 110 may include various devices, such as a controller, an image preprocessor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, a memory, or any other type of device for image processing and analysis. The image preprocessor may include a video processor for capturing, digitizing, and processing images from an image sensor. The CPU may include any number of microcontrollers or microprocessors. The support circuits may be any number of circuits generally known in the art, including caches, power supplies, clocks, and input-output circuits. The memory may store software that controls the operation of the system when executed by the processor. The memory may include a database and image processing software. The memory may include any number of random access memories, read-only memories, flash memories, disk drives, optical storage media, tape storage media, removable storage media, and other types of storage media. In one example, the memory may be separated from the processing unit 110. In another example, the memory may be integrated into the processing unit 110.

[0103] Each memory 140, 150 may include software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), may control the operation of various aspects of system 100. These memory units may include various databases and image processing software. The memory units may include random access memory, read-only memory, flash memory, disk drives, optical storage media, tape storage media, removable storage media, and / or any other type of storage media. In some embodiments, the memory units 140, 150 may be separate from the application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into the application processor 180 and / or image processor 190.

[0104] Position sensor 130 may include any type of device suitable for determining a location associated with at least one component of system 100. In some embodiments, position sensor 130 may include a GPS receiver. Such a receiver may determine user location and velocity by processing signals propagated by global positioning system satellites. Position information from position sensor 130 may be used by application processor 180 and / or image processor 190.

[0105] The user interface 170 may include any device suitable for providing information to or for receiving input from one or more users of the system 100. In some embodiments, the user interface 170 may include a user input device including, for example, a touch screen, a microphone, a keyboard, a pointer device, a tracking wheel, a camera, a knob, a button, etc. In the case of such an input device, the user can provide information input or commands to the system 100 by typing instructions or information, providing voice commands, selecting menu options on a screen using buttons, a pointer or eye tracking functionality, or any other suitable technique for communicating information to the system 100.

[0106] The user interface 170 may be equipped with one or more processing devices configured to provide information to or receive information from a user and process the information for use, for example, by the application processor 180. In some embodiments, such a processing device may execute instructions to recognize and track eye movements, receive and interpret voice commands, recognize and interpret touches and / or gestures made on a touch screen, respond to keyboard input or menu selections, etc. In some embodiments, the user interface 170 may include a display, a speaker, a haptic device, and / or any other device for providing output information to a user.

[0107] The map database 160 may include any type of database for storing map data useful to the system 100. In some embodiments, the map database 160 may include data associated with locations in a reference coordinate system for various items, including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, and the like. The map database 160 may store not only the locations of such items, but also descriptive terms associated with such items (including, for example, names associated with any stored features). In some embodiments, the map database 160 may be physically located with other components of the system 100. Alternatively or in addition, the map database 160 or a portion thereof may be remotely located relative to other components of the system 100 (e.g., the processing unit 110). In such embodiments, information from the map database 160 may be downloaded via a wired or wireless data connection of a network (e.g., via a cellular network and / or the Internet, etc.).

[0108] Image capture devices 122, 124, and 126 may each include any type of device suitable for capturing at least one image from an environment. Moreover, any number of image capture devices may be used to acquire images for input to the image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, or even more than four image capture devices. Image capture devices 122, 124, and 126 will be referred to below. FIG. 2B to FIG. 2E Further description.

[0109] System 100 or its various components may be integrated into a variety of different platforms. In some embodiments, system 100 may include Figure 2A On the vehicle 200 shown. For example, as shown above with respect to Figure 1 As described above, vehicle 200 may be equipped with processing unit 110 and any other components of system 100. In some embodiments, vehicle 200 may be equipped with only a single image capture device (e.g., a camera). FIG. 2B to FIG. 2E In other embodiments discussed herein, multiple image capture devices may be used. Figure 2A Either of the image capture devices 122 and 124 of the illustrated vehicle 200 may be part of an ADAS (Advanced Driver Assistance System) imaging suite.

[0110] The image capture device included on the vehicle 200 as part of the image acquisition unit 120 may be located at any suitable location. FIG. 2A to FIG. 2E and FIG. 3A to FIG. 3CAs shown, the image capture device 122 may be located near the rearview mirror. This location may provide a line of sight (similar to the line of sight of the driver of the vehicle 200) that may help determine what is visible and what is not visible to the driver. The image capture device 122 may be positioned anywhere near the rearview mirror, but placing the image capture device 122 on the driver's side of the mirror may further help obtain an image representative of the driver's field of view and / or line of sight.

[0111] Other locations may also be used for the image capture device of the image acquisition unit 120. For example, the image capture device 124 may be located on or in the bumper of the vehicle 200. Such a location may be particularly suitable for an image capture device with a wide field of view. The line of sight of an image capture device located on the bumper may be different from the line of sight of the driver, and therefore, the bumper image capture device and the driver may not always see the same object. Image capture devices (e.g., image capture devices 122, 124, and 126) may also be located in other locations. For example, the image capture device may be located on or in one or both side mirrors of the vehicle 200, on the roof of the vehicle 200, on the hood of the vehicle 200, on the trunk of the vehicle 200, on the side of the vehicle 200, mounted on any window of the vehicle 200, positioned behind or in front of any window of the vehicle 200, and mounted in or near the light pattern on the front and / or back of the vehicle 200, and so on.

[0112] In addition to the map capture device, the vehicle 200 may include various other components of the system 100. For example, the processing unit 110 may be included on the vehicle 200, either integrated or separate from the vehicle's engine control unit (ECU). The vehicle 200 may also be equipped with a location sensor 130, such as a GPS receiver, and may also include a map database 160 and memory units 140 and 150.

[0113] Figure 2A is a diagrammatic side view representation of an exemplary vehicle imaging system consistent with disclosed embodiments. Figure 2B yes Figure 2A A schematic top view of the embodiment shown. Figure 2B As illustrated in , the disclosed embodiments may include a vehicle 200 including a system 100 in its body having a first image capture device 122 positioned near a rearview mirror and / or near a driver of the vehicle 200, a second image capture device 124 positioned on or in a bumper area (e.g., one of the bumper areas 210) of the vehicle 200, and a processing unit 110.

[0114] like Figure 2CAs illustrated in FIG. 1 , both image capture devices 122 and 124 may be positioned near a rearview mirror and / or near the driver of vehicle 200. Figure 2B and Figure 2C Two image capture devices 122 and 124 are shown in FIG, but it should be understood that other embodiments may include more than two image capture devices. Figure 2D and Figure 2E In the illustrated embodiment, a first image capture device 122 , a second image capture device 124 , and a third image capture device 126 are included in the system 100 of a vehicle 200 .

[0115] like Figure 2D As illustrated in FIG. 1 , image capture device 122 may be positioned near a rearview mirror and / or near a driver of vehicle 200, and image capture devices 124 and 126 may be positioned on or in a bumper region (e.g., one of bumper regions 210) of vehicle 200. Figure 2E As shown, image capture devices 122, 124, and 126 may be positioned near rearview mirrors and / or near the driver's seat of vehicle 200. The disclosed embodiments are not limited to any particular number and configuration of image capture devices, and the image capture devices may be positioned in any suitable locations in and / or on vehicle 200.

[0116] It should be understood that the disclosed embodiments are not limited to vehicles and can be applied in other contexts. It should also be understood that the disclosed embodiments are not limited to a specific type of vehicle 200 and can be applied to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.

[0117] The first image capture device 122 may include any suitable type of image capture device. The image capture device 122 may include an optical axis. In one example, the image capture device 122 may include an Aptina M9V024 WVGA sensor with a global shutter. In other embodiments, the image capture device 122 may provide a resolution of 1280x960 pixels and may include a rolling shutter. The image capture device 122 may include various optical elements. In some embodiments, for example, one or more lenses may be included to provide the image capture device with a desired focal length and field of view. In some embodiments, the image capture device 122 may be associated with a 6 mm lens or a 12 mm lens. In some embodiments, such as Figure 2D, the image capture device 122 may be configured to acquire images having a desired field of view (FOV) 202. For example, the image capture device 122 may be configured to have a conventional FOV, such as in the range of 40 degrees to 56 degrees, including 46 degree FOV, 50 degree FOV, 52 degree FOV or more. Alternatively, the image capture device 122 may be configured to have a narrow FOV in the range of 23 degrees to 40 degrees, such as a 28 degree FOV or a 36 degree FOV. In addition, the image capture device 122 may be configured to have a wide FOV in the range of 100 degrees to 180 degrees. In some embodiments, the image capture device 122 may include a wide angle bumper camera or have a FOV of up to 180 degrees.

[0118] The first image capture device 122 may acquire a plurality of first images relative to a scene associated with the vehicle 200. Each of the plurality of first images may be acquired as a series of image scan lines that may be captured using a rolling shutter. Each scan line may include a plurality of pixels.

[0119] The first image capture device 122 may have a scan rate associated with acquiring each of the first series of image scan lines. The scan rate may refer to the rate at which the image sensor may acquire image data associated with each pixel included in a particular scan line.

[0120] For example, image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, including CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be employed in conjunction with a rolling shutter so that each pixel in a row is read one at a time, and the scanning of the row is performed on a row-by-row basis until the entire image frame has been captured. In some embodiments, the rows may be captured sequentially from top to bottom with respect to the frame.

[0121] Using a rolling shutter can cause pixels in different rows to be exposed and captured at different times, which can cause skew and other image artifacts in the captured image frame. On the other hand, when image capture device 122 is configured to operate with a global or synchronized shutter, all pixels can be exposed for the same amount of time and in a common exposure period. As a result, the image data of a frame collected from a system employing a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a specific time. In contrast, in a rolling shutter application, each row in the frame is exposed and the data is captured at a different time. As a result, moving objects may appear distorted in an image capture device with a rolling shutter. This phenomenon will be described in more detail below.

[0122] The second image capture device 124 and the third image capture device 126 may be any type of image capture device. As with the first image capture device 122, each of the image capture devices 124 and 126 may include an optical axis. In one embodiment, each of the image capture devices 124 and 126 may include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of the image capture devices 124 and 126 may include a rolling shutter. As with the image capture device 122, the image capture devices 124 and 126 may be configured to include various lenses and optical elements. In some embodiments, the lenses associated with the image capture devices 124 and 126 may provide a FOV (such as FOVs 204 and 206) that is equal to or narrower than the FOV (such as FOV 202) associated with the image capture device 122. For example, the image capture devices 124 and 126 may have a FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less.

[0123] Image capture devices 124 and 126 may acquire a plurality of second and third images relative to a scene associated with vehicle 200. Each of the plurality of second and third images may be acquired as a second series and a third series of image scan lines that may be captured using a rolling shutter. Each scan line or row may have a plurality of pixels. Image capture devices 124 and 126 may have a second scan rate and a third scan rate associated with acquiring each image scan line included in the second series and the third series.

[0124] Each image capture device 122, 124, and 126 may be positioned at any suitable location and orientation relative to vehicle 200. The relative positioning of image capture devices 122, 124, and 126 may be selected to help fuse together information obtained from the image capture devices. For example, in some embodiments, a FOV associated with image capture device 124 (such as FOV 204) may partially or completely overlap with a FOV associated with image capture device 122 (such as FOV 202) and a FOV associated with image capture device 126 (such as FOV 206).

[0125] Image capture devices 122, 124, and 126 may be located at any suitable relative heights on vehicle 200. In one example, there may be height differences between image capture devices 122, 124, and 126 that may provide sufficient parallax information to enable stereo analysis. Figure 2A As shown, the two image capture devices 122 and 124 are at different heights. For example, there may also be lateral displacement differences between the image capture devices 122, 124, and 126, providing additional parallax information for the stereo analysis of the processing unit 110. The lateral displacement difference can be denoted as d x ,like Figure 2C and Figure 2D In some embodiments, a fore-and-aft displacement (e.g., a range displacement) may exist between image capture devices 122, 124, and 126. For example, image capture device 122 may be located 0.5 meters to more than 2 meters behind image capture device 124 and / or image capture device 126. This type of displacement may enable one of the image capture devices to cover a potential blind spot of the other image capture devices.

[0126] Image capture device 122 may have any suitable resolution capabilities (e.g., the number of pixels associated with the image sensor), and the resolution of the image sensor associated with image capture device 122 may be greater than, less than, or equal to the resolution of the image sensors associated with image capture devices 124 and 126. In some embodiments, the image sensors associated with image capture device 122 and / or image capture devices 124 and 126 may have a resolution of 640 x 480, 1024 x 768, 1280 x 960, or any other suitable resolution.

[0127] The frame rate (e.g., the rate at which an image capture device acquires a set of pixel data for one image frame before moving to capture pixel data associated with the next image frame) may be controllable. The frame rate associated with image capture device 122 may be greater than, less than, or equal to the frame rate associated with image capture devices 124 and 126. The frame rate associated with image capture devices 122, 124, and 126 may depend on various factors that may affect the timing of the frame rate. For example, one or more of image capture devices 122, 124, and 126 may include an optional pixel delay period imposed before or after acquiring image data associated with one or more pixels of an image sensor in image capture devices 122, 124, and / or 126. In general, image data corresponding to each pixel may be acquired according to the clock rate of the device (e.g., one pixel per clock cycle). Additionally, in embodiments including a rolling shutter, one or more of the image capture devices 122, 124, and 126 may include an optional horizontal blanking period imposed before or after acquiring image data associated with a row of pixels of an image sensor in the image capture devices 122, 124, and / or 126. Further, one or more of the image capture devices 122, 124, and / or 126 may include an optional vertical blanking period imposed before or after acquiring image data associated with an image frame of the image capture devices 122, 124, and 126.

[0128] These timing controls may synchronize the frame rates associated with image capture devices 122, 124, and 126, even when the line scan rate of each image capture device is different. Additionally, as will be discussed in more detail below, among other factors (e.g., image sensor resolution, maximum line scan rate, etc.), these optional timing controls may synchronize image capture of areas where the FOV of image capture device 122 overlaps with one or more FOVs of image capture devices 124 and 126, even when the field of view of image capture device 122 is different from the FOVs of image capture devices 124 and 126.

[0129] The frame rate timing of image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensors. For example, assuming that the two devices have similar line scan rates, if one device includes an image sensor with a resolution of 640 x 480 and the other device includes an image sensor with a resolution of 1280 x 960, more time will be required to acquire a frame of image data from the sensor with the higher resolution.

[0130] Another factor that may affect the timing of image data acquisition for image capture devices 122, 124, and 126 is the maximum line scan rate. For example, it will take some minimum amount of time to acquire a row of image data for the image sensors included in image capture devices 122, 124, and 126. Assuming no pixel delay period is added, this minimum amount of time to acquire a row of image data will be related to the maximum line scan rate of the particular device. Devices that provide higher maximum line scan rates potentially provide greater frame rates than devices with smaller maximum line scan rates. In some embodiments, the maximum line scan rate of one or more of image capture devices 124 and 126 may be greater than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture devices 124 and / or 126 may be 1.25 times, 1.5 times, 1.75 times, or more than 2 times the maximum line scan rate of image capture device 122.

[0131] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may operate at a scan rate less than or equal to its maximum scan rate. The system may be configured such that one or more of image capture devices 124 and 126 operate at a line scan rate equal to the line scan rate of image capture device 122. In other examples, the system may be configured such that the line scan rate of image capture device 124 and / or image capture device 126 may be 1.25 times, 1.5 times, 1.75 times, or more than 2 times the line scan rate of image capture device 122.

[0132] In some embodiments, the image capture devices 122, 124, and 126 may be asymmetric. That is, they may include cameras with different fields of view (FOV) and focal lengths. For example, the fields of view of the image capture devices 122, 124, and 126 may include any desired area relative to the environment of the vehicle 200. In some embodiments, one or more of the image capture devices 122, 124, and 126 may be configured to acquire image data from the environment in front of the vehicle 200, behind the vehicle 200, to the side of the vehicle 200, or a combination thereof.

[0133] Further, the focal length associated with each image capture device 122, 124, and / or 126 may be selectable (e.g., by including an appropriate lens, etc.) so that each device captures images of objects at a desired range of distances relative to the vehicle 200. For example, in some embodiments, the image capture devices 122, 124, and 126 may capture images of close-up objects within a few meters from the vehicle. The image capture devices 122, 124, and 126 may also be configured to capture images of objects at a range farther from the vehicle (e.g., 25m, 50m, 100m, 150m or more). Further, the focal lengths of image capture devices 122, 124, and 126 may be selected so that one image capture device (e.g., image capture device 122) can capture images of objects that are relatively close to the vehicle (e.g., within 10 m or within 20 m), while other image capture devices (e.g., image capture devices 124 and 126) can capture images of objects that are farther from the vehicle 200 (e.g., greater than 20 m, 50 m, 100 m, 150 m, etc.).

[0134] According to some embodiments, the FOV of one or more of the image capture devices 122, 124, and 126 may have a wide angle. For example, it may be advantageous to have a FOV of 140 degrees, particularly for image capture devices 122, 124, and 126 that may be used to capture images of an area near the vehicle 200. For example, the image capture device 122 may be used to capture images of an area to the right or left of the vehicle 200, and in such embodiments, it may be desirable for the image capture device 122 to have a wide FOV (e.g., at least 140 degrees).

[0135] The field of view associated with each of the image capture devices 122, 124, and 126 may depend on the corresponding focal length. For example, as the focal length increases, the corresponding field of view decreases.

[0136] Image capture devices 122, 124, and 126 may be configured to have any suitable field of view. In one particular example, image capture device 122 may have a horizontal FOV of 46 degrees, image capture device 124 may have a horizontal FOV of 23 degrees, and image capture device 126 may have a horizontal FOV between 23 degrees and 46 degrees. In another example, image capture device 122 may have a horizontal FOV of 52 degrees, image capture device 124 may have a horizontal FOV of 26 degrees, and image capture device 126 may have a horizontal FOV between 26 degrees and 52 degrees. In some embodiments, the ratio of the FOV of image capture device 122 to the FOV of image capture device 124 and / or image capture device 126 may vary from 1.5 to 2.0. In other embodiments, the ratio may vary between 1.25 and 2.25.

[0137] The system 100 may be configured such that the field of view of the image capture device 122 at least partially or completely overlaps the field of view of the image capture device 124 and / or the image capture device 126. In some embodiments, the system 100 may be configured such that the fields of view of the image capture devices 124 and 126, for example, fall within (e.g., are narrower than) the field of view of the image capture device 122 and share a common center with the field of view of the image capture device 122. In other embodiments, the image capture devices 122, 124, and 126 may capture adjacent FOVs, or may have their FOVs partially overlap. In some embodiments, the fields of view of the image capture devices 122, 124, and 126 may be aligned such that the center of the narrower FOV image capture device 124 and / or 126 may be located in the lower half of the field of view of the wider FOV device 122.

[0138] Figure 2F is a schematic representation of an exemplary vehicle control system consistent with the disclosed embodiments. Figure 2FAs indicated in , vehicle 200 may include throttle system 220, braking system 230, and steering system 240. System 100 may provide input (e.g., control signals) to one or more of throttle system 220, braking system 230, and steering system 240 via one or more data links (e.g., any wired and / or wireless one or more links for transmitting data). For example, based on analysis of images acquired by image capture devices 122, 124, and / or 126, system 100 may provide control signals to one or more of throttle system 220, braking system 230, and steering system 240 to navigate vehicle 200 (e.g., by causing acceleration, turning, lane shifting, etc.). Further, system 100 may receive input from one or more of throttle system 220, braking system 230, and steering system 240 indicating an operating condition of vehicle 200 (e.g., speed, whether vehicle 200 is braking and / or turning, etc.). Further details are provided below in conjunction with Figures 4 to 7 To provide.

[0139] like Figure 3A As shown, the vehicle 200 may also include a user interface 170 for interacting with a driver or passenger of the vehicle 200. For example, in a vehicle application, the user interface 170 may include a touch screen 320, a knob 330, a button 340, and a microphone 350. The driver or passenger of the vehicle 200 may also use a handle (e.g., located on or near the steering column of the vehicle 200, such as a turn signal handle), a button (e.g., located on the steering wheel of the vehicle 200), etc. to interact with the system 100. In some embodiments, the microphone 350 may be positioned adjacent to the rearview mirror 310. Similarly, in some embodiments, the image capture device 122 may be located near the rearview mirror 310. In some embodiments, the user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, the system 100 may provide various notifications (e.g., prompts) via the speaker 360.

[0140] FIG. 3B to FIG. 3D is an illustration of an exemplary camera mount 370 that is configured to be positioned behind a rearview mirror (e.g., rearview mirror 310) and against a vehicle windshield, consistent with disclosed embodiments. Figure 3B As shown, camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be positioned behind visor 380, which may be flush with the vehicle windshield and include a combination of film and / or anti-reflective material. For example, visor 380 may be positioned so that it is aligned against a vehicle windshield having a matching slope. In some embodiments, for example, Figure 3D, each of the image capture devices 122, 124, and 126 may be positioned behind a light shield 380. The disclosed embodiments are not limited to any particular configuration of the image capture devices 122, 124, and 126, the camera mounts 370, and the light shield 380. Figure 3C yes Figure 3B 0 is an illustration of the camera mount 370 from a front view.

[0141] As will be appreciated by those skilled in the art having the benefit of this disclosure, many variations and / or modifications may be made to the foregoing disclosed embodiments. For example, not all components are essential to the operation of the system 100. Further, any component may be located in any appropriate portion of the system 100, and the components may be rearranged into various configurations while providing the functionality of the disclosed embodiments. Thus, the foregoing configurations are examples and, independent of the configurations discussed above, the system 100 may provide a wide range of functionality for analyzing the surrounding environment of the vehicle 200 and navigating the vehicle 200 in response to the analysis.

[0142] As discussed in further detail below and consistent with various disclosed embodiments, the system 100 can provide various features related to autonomous driving and / or driver assistance technology. For example, the system 100 can analyze image data, location data (e.g., GPS location information), map data, speed data, and / or data from sensors included in the vehicle 200. The system 100 can collect data from, for example, the image acquisition unit 120, the location sensor 130, and other sensors for analysis. Further, the system 100 can analyze the collected data to determine whether the vehicle 200 should take certain actions, and then automatically take the determined actions without human intervention. For example, when the vehicle 200 is navigating without human intervention, the system 100 can automatically control the braking, acceleration, and / or steering of the vehicle 200 (e.g., by sending control signals to one or more of the throttle system 220, the braking system 230, and the steering system 240). Further, the system 100 can analyze the collected data and issue warnings, and / or prompt vehicle occupants based on the analysis of the collected data. Additional details about various embodiments provided by the system 100 are provided below.

[0143] Forward-facing multi-imaging system

[0144] As discussed above, the system 100 may provide driver assistance functionality using a multi-camera system. - The multi-camera system may use one or more cameras facing the forward direction of the vehicle. In other embodiments, the multi-camera system may include one or more cameras facing the side of the vehicle or the rear of the vehicle. In one embodiment, for example, the system 100 may use a dual-camera imaging system, wherein a first camera and a second camera (e.g., image capture devices 122 and 124) may be positioned in front of and / or on the side of a vehicle (e.g., vehicle 200). The field of view of the first camera may be larger, smaller, or partially overlap the field of view of the second camera. In addition, the first camera may be connected to a first image processor to perform a monocular image analysis of the image provided by the first camera, and the second camera may be connected to a second image processor to perform a monocular image analysis of the image provided by the second camera. The outputs (e.g., processed information) of the first image processor and the second image processor may be combined. In some embodiments, the second image processor may receive images from both the first camera and the second camera to perform stereo analysis. In another embodiment, the system 100 may use a three-camera imaging system, wherein each camera has a different field of view. Thus, such a system may make decisions based on information obtained from objects located at varying distances both in front of and to the sides of the vehicle. References to monocular image analysis may refer to instances where image analysis is performed based on images captured from a single viewpoint (e.g., from a single camera). Stereoscopic image analysis may refer to instances where image analysis is performed based on two or more images captured with one or more variations of image capture parameters. For example, captured images suitable for performing stereoscopic image analysis may include images captured from two or more different positions, from different fields of view, using different focal lengths, incorporating parallax information, and the like.

[0145] For example, in one embodiment, the system 100 may implement a three-camera configuration using image capture devices 122-126. In such a configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees, or other values ​​selected from a range of about 20 degrees to 45 degrees, etc.), image capture device 124 may provide a wide field of view (e.g., 150 degrees or other values ​​selected from a range of about 100 degrees to about 180 degrees), and image capture device 126 may provide an intermediate field of view (e.g., 46 degrees or other values ​​selected from a range of about 35 degrees to about 60 degrees). In some embodiments, image capture device 126 may serve as a primary or main camera. Image capture devices 122-126 may be positioned behind rearview mirror 310 and positioned substantially side by side (e.g., 6 cm apart). Further, in some embodiments, as discussed above, one or more of image capture devices 122-126 may be mounted behind a sun visor 380 that is flush with the windshield of vehicle 200. Such shading minimizes the effect of any reflections from the interior of the vehicle on the image capture devices 122-126.

[0146] In another embodiment, as above combined with Figure 3B and Figure 3C As discussed, a wide field of view camera (e.g., image capture device 124 in the above example) can be mounted lower than a narrow primary field of view camera (e.g., image devices 122 and 126 in the above example). This configuration can provide a free line of sight from the wide field of view camera. To reduce reflections, the camera can be mounted close to the windshield of the vehicle 200, and a polarizer on the camera can be included to suppress reflected light.

[0147] A three-camera system may provide certain performance characteristics. For example, some embodiments may include the ability to verify detection of an object by one camera based on detection results of another camera. - In the three-camera configuration discussed above, the processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips, as discussed above), each processing device dedicated to processing images captured by one or more of the image capture devices 122-126.

[0148] In a three-camera system, the first processing device may receive images from both the main camera and the narrow field of view camera, and perform visual processing of the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Further, the first processing device may calculate the pixel difference between the images of the main camera and the narrow camera and create a three-dimensional reconstruction of the environment of the vehicle 200. The first processing device may then combine the three-dimensional reconstruction with three-dimensional map data, or with three-dimensional information calculated based on information from another camera.

[0149] The second processing device may receive images from the primary camera and perform visual processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. In addition, the second processing device may calculate camera displacement, and based on the displacement, calculate pixel differences between consecutive images and create a 3D reconstruction of the scene (e.g., structure from motion). The second processing device may send the structure from motion-based 3D reconstruction to the first processing device to be combined with the stereoscopic 3D image.

[0150] The third processing device may receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device may further execute additional processing instructions to analyze the images to identify objects moving in the images, such as vehicles changing lanes, pedestrians, and the like.

[0151] In some embodiments, having image-based information streams captured and processed independently can provide an opportunity to provide system redundancy. For example, such redundancy can include using a first image capture device and images processed from that device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.

[0152] In some embodiments, the system 100 may use two image capture devices (e.g., image capture devices 122 and 124) to provide navigation assistance for the vehicle 200, and use a third image capture device (e.g., image capture device 126) to provide redundancy and verify the analysis of data received from the other two image capture devices. For example, in such a configuration, the image capture devices 122 and 124 may provide images for stereo analysis of the system 100 to navigate the vehicle 200, while the image capture device 126 may provide images for monocular analysis of the system 100 to provide redundancy and verify information obtained based on images captured from the image capture devices 122 and / or the image capture devices 124. That is, the image capture device 126 (and corresponding processing device) may be considered to provide a redundant subsystem to provide a check on the analysis obtained from the image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AEB) system).

[0153] Those skilled in the art will recognize that the above camera configurations, camera placements, number of cameras, camera locations, etc. are merely examples. These components and other components described with respect to the overall system may be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. The following are further details regarding the use of a multi-camera system to provide driver assistance and / or autonomous vehicle functionality.

[0154] Figure 41 is an exemplary functional block diagram of memory 140 and / or 150 consistent with the disclosed embodiments, which memory 140 and / or 150 may store / programmed with instructions for performing one or more operations. Although the following is referenced to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0155] like Figure 4 As shown, the memory 140 may store a monocular image analysis module 402, a stereo image analysis module 404, a velocity and acceleration module 406, and a navigation response module 408. The disclosed embodiments are not limited to any particular configuration of the memory 140. - Further, the application processor 180 and / or the image processor 190 may execute instructions stored in any of the modules 402-408 included in the memory 140. It will be appreciated by those skilled in the art that references to the processing unit 110 discussed below may refer individually or collectively to the application processor 180 and the image processor 190. Therefore, any of the following processing steps may be performed by one or more processing devices.

[0156] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform a monocular image analysis of a set of images acquired by one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensory information (e.g., information from a radar) to perform the monocular image analysis. As described below in conjunction with FIG. 5A to FIG. 5D As described, the monocular image analysis module 402 may include instructions for detecting a set of features within a set of images (such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other features associated with the vehicle environment. Based on this analysis, the system 100 (e.g., via the processing unit 110) may generate one or more navigation responses in the vehicle 200, such as turns, lane shifts, acceleration changes, etc., which are discussed below in conjunction with the navigation response module 408.

[0157] In one embodiment, stereo image analysis module 404 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform stereo image analysis of a first set of second sets of images acquired by a combination of image capture devices selected from any of image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from the first set of second sets of images with additional sensory information (e.g., information from radar) to perform stereo image analysis. For example, stereo image analysis module 404 may include instructions for performing stereo image analysis based on a first set of images acquired by image capture device 124 and a second set of images acquired by image capture device 126. As described below in conjunction with Figure 6 As described, stereo image analysis module 404 may include instructions for detecting a set of features within the first and second sets of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, etc. Based on this analysis, processing unit 110 may generate one or more navigation responses in vehicle 200, such as turns, lane shifts, acceleration changes, etc., as discussed below in conjunction with navigation response module 408.

[0158] In one embodiment, the speed and acceleration module 406 may store software configured to analyze data received from one or more computer-electronic devices of the vehicle 200 that are configured to cause a change in the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 may execute instructions associated with the speed and acceleration module 406 to calculate a target speed for the vehicle 200 based on data resulting from executing the monocular image analysis module 402 and / or the stereo image analysis module 404. For example, such data may include: target position, speed, and / or acceleration; position and / or speed of the vehicle 200 relative to nearby vehicles, pedestrians, or road objects; position information of the vehicle 200 relative to lane markings of the road, etc. In addition, the processing unit 110 may calculate the target speed for the vehicle 200 based on sensory input (e.g., information from a radar) and input from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and / or the steering system 240 of the vehicle 200. Based on the calculated target speed, the processing unit 110 may send electronic signals to the throttle system 220, the braking system 230 and / or the steering system 240 of the vehicle 200 to trigger a change in speed and / or acceleration by, for example, physically pressing the brakes or releasing the accelerator of the vehicle 200.

[0159] In one embodiment, the navigation response module 408 may store software executable by the processing unit 110 to determine a desired navigation response based on data resulting from execution of the monocular image analysis module 402 and / or the stereoscopic image analysis module 404. Such data may include position and velocity information associated with nearby vehicles, pedestrians, and road objects, target position information for the vehicle 200, and the like. Additionally, in some embodiments, the navigation response may be based (partially or entirely) on map data, a predetermined position of the vehicle 200, and / or a relative velocity or relative acceleration between the vehicle 200 and one or more objects detected from execution of the monocular image analysis module 402 and / or the stereoscopic image analysis module 404. The navigation response module 408 may also determine the desired navigation response based on sensory input (e.g., information from a radar) and input from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and the steering system 240 of the vehicle 200. Based on the desired navigation response, the processing unit 110 may send electronic signals to the throttle system 220, the brake system 230, and the steering system 240 of the vehicle 200 to trigger the desired navigation response, for example, by turning the steering wheel of the vehicle 200 to achieve a predetermined angle of rotation. In some embodiments, the processing unit 110 may use the output of the navigation response module 408 (e.g., the desired navigation response) as an input to execute the speed and acceleration module 406 to calculate the speed change of the vehicle 200.

[0160] Figure 5A is a flow chart illustrating an exemplary process 500A for generating one or more navigation responses based on monocular image analysis consistent with disclosed embodiments. At step 510, the processing unit 110 may receive a plurality of images via a data interface 128 located between the processing unit 110 and the image acquisition unit 120. For example, a camera (such as an image capture device 122 having a field of view 202) included in the image acquisition unit 120 may capture a plurality of images of an area in front of the vehicle 200 (e.g., or to the side or rear of the vehicle) and send the images to the processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). At step 520, the processing unit 110 may execute the monocular image analysis module 402 to analyze the plurality of images, which is described below in conjunction with the monocular image analysis module 402. FIG. 5B to FIG. 5D By performing the analysis, processing unit 110 may detect a set of features within a set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, etc.

[0161] At step 520, the processing unit 110 may also execute the monocular image analysis module 402 to detect various road hazards, such as a portion of a truck tire, a fallen road sign, loose cargo, small animals, etc. Road hazards may vary in structure, shape, size, and color, which may make detecting such hazards more challenging. In some embodiments, the processing unit 110 may execute the monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, the processing unit 110 may estimate the camera motion between consecutive image frames and calculate the pixel gap between frames to construct a three-dimensional map of the road. The processing unit 110 may then use the three-dimensional map to detect the road surface and hazards that exist above the road surface.

[0162] At step 530, the processing unit 110 may execute the navigation response module 408 to perform the following operations based on the analysis performed at step 520 and the above combined Figure 4 The techniques described above may be used to generate one or more navigation responses of the vehicle 200. For example, the navigation response may include a turn, a lane shift, an acceleration change, etc. In some embodiments, the processing unit 110 may use data obtained from executing the speed and acceleration module 406 to generate one or more navigation responses. In addition, multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof. For example, the processing unit 110 may cause the vehicle 200 to change a lane and then accelerate by, for example, sending control signals to the steering system 240 and the throttle system 220 of the vehicle 200 in sequence. Alternatively, the processing unit 110 may cause the vehicle 200 to brake while changing lanes by, for example, sending control signals to the braking system 230 and the steering system 240 of the vehicle 200 at the same time.

[0163] Figure 5B is a flow chart showing an exemplary process 500B for detecting one or more vehicles and / or pedestrians in a set of images consistent with the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500B. At step 540, processing unit 110 may determine a set of candidate objects representing possible vehicles and / or pedestrians. For example, processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify possible locations within each image that may contain objects of interest (e.g., vehicles, pedestrians, or portions thereof). The predetermined patterns may be designed in a manner that achieves a high rate of "false hits" and a low rate of "misses." For example, processing unit 110 may use a low threshold similarity to a predetermined pattern to identify a candidate object as a possible vehicle or pedestrian. Doing so may allow processing unit 110 to reduce the probability of missing (e.g., not identifying) a candidate object representing a vehicle or pedestrian.

[0164] At step 542, processing unit 110 may filter the set of candidate objects to exclude certain candidates (e.g., irrelevant or less relevant objects) based on classification criteria. Such criteria may be derived from various attributes associated with object types stored in a database (e.g., a database stored in memory 140). Attributes may include object shape, size, texture, position (e.g., relative to vehicle 200), etc. Thus, processing unit 110 may use one or more sets of criteria to reject erroneous candidates from a set of candidate objects.

[0165] At step 544, processing unit 110 may analyze multiple frames of the image to determine whether an object in a set of candidate objects represents a vehicle and / or a pedestrian. For example, processing unit 110 may track detected candidate objects across consecutive frames and accumulate frame-by-frame data associated with the detected objects (e.g., size, position relative to vehicle 200, etc.). In addition, processing unit 110 may estimate parameters of the detected objects and compare the frame-by-frame position data of the objects with the predicted positions.

[0166] At step 546, the processing unit 110 may construct a set of measurements of the detected object. For example, such measurements may include position, velocity, and acceleration values ​​(relative to the vehicle 200) associated with the detected object. In some embodiments, the processing unit 110 may construct the measurements based on the following techniques: using a series of time-based observations based on estimation techniques (such as Kalman filters or linear quadratic estimation (LQE)); and / or based on available modeling data for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). The Kalman filter may be based on measurements of object scale, where the scale measurement is proportional to the time to collision (e.g., the amount of time it takes for the vehicle 200 to reach the object). Thus, by executing steps 540-546, the processing unit 110 may identify vehicles and pedestrians that appear within a set of captured images and obtain information associated with the vehicles and pedestrians (e.g., position, velocity, size). Based on this identification and the information obtained, the processing unit 110 may generate one or more navigation responses for the vehicle 200, which is combined with the above Figure 5A describe.

[0167] At step 548, the processing unit 110 may perform an optical flow analysis of one or more images to reduce the probability of detecting "false hits" and missing candidate objects representing vehicles or pedestrians. Optical flow analysis may refer to, for example, analyzing the motion pattern of the vehicle 200 in one or more images associated with other vehicles and pedestrians, and distinguishing from road motion. The processing unit 110 may calculate the motion of the candidate objects by observing the different positions of the objects across multiple image frames (captured at different times). The processing unit 110 may use the position and time values ​​as inputs to a mathematical model for calculating the motion of the candidate objects. Thus, optical flow analysis may provide another method for detecting vehicles and pedestrians near the vehicle 200. The processing unit 110 may perform optical flow analysis in combination with steps 540-546 to provide redundancy for detecting vehicles and pedestrians and increase the reliability of the system 100.

[0168] Figure 5C is a flow chart illustrating an exemplary process 500C for detecting road signs and / or lane geometry information in a set of images consistent with disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500C. At step 550, processing unit 110 may detect a set of objects by scanning one or more images. In order to detect a section of lane markings, lane geometry information, and other related road signs, processing unit 110 may filter a set of objects to exclude those objects that are determined to be irrelevant (e.g., minor potholes, small rocks, etc.). At step 552, processing unit 110 may group together the segments that belong to the same road sign or lane marking detected in step 550. Based on the grouping, processing unit 110 may establish a model, such as a mathematical model, representing the detected segments.

[0169] At step 554, the processing unit 110 may construct a set of measurements associated with the detected segment. - In some embodiments, the processing unit 110 may create a projection of the detected segment from the image plane to the real world plane. The projection may be characterized using a cubic polynomial having coefficients corresponding to physical properties such as position, slope, curvature, and curvature derivatives of the detected road. When generating the projection, the processing unit 110 may take into account changes in the road surface, as well as the pitch and yaw rates associated with the vehicle 200. In addition, the processing unit 110 may model the road elevation by analyzing the position and motion cues present on the road surface. Further, the processing unit 110 may estimate the pitch and yaw rates associated with the vehicle 200 by tracking a set of feature points in one or more images.

[0170] At step 556, processing unit 110 may perform a multi-frame analysis by, for example, tracking detected segments across consecutive image frames and accumulating frame-by-frame data associated with the detected segments. As processing unit 110 performs a multi-frame analysis, the set of measurements constructed at step 554 may become more reliable and associated with a higher confidence level. Thus, by performing steps 550-556, processing unit 110 may identify road signs that appear within a set of captured images and derive lane geometry information. Based on this identification and derived information, processing unit 110 may generate one or more navigation responses for vehicle 200, which is combined with the above Figure 5A describe.

[0171] At step 558, processing unit 110 may consider additional information sources to further build a safety model of vehicle 200 in the context of its surroundings. Processing unit 110 may use the safety model to define the context in which system 100 may exercise autonomous control of vehicle 200 in a safe manner. To build the safety model, in some embodiments, processing unit 110 may consider the positions and motions of other vehicles, roadsides and obstacles of the detected road, and / or an overall road shape description extracted from map data (such as data from map database 160). By considering additional information sources, processing unit 110 may provide redundancy to detect road signs and lane geometry and increase the reliability of system 100.

[0172] Figure 5D 5 is a flow chart illustrating an exemplary process 500D for detecting a traffic light in a set of images consistent with disclosed embodiments. The processing unit 110 may execute the monocular image analysis module 402 to implement the process 500D. At step 560, the processing unit 110 may scan a set of images and identify objects that appear at locations in the images that may contain traffic lights. For example, the processing unit 110 may filter the identified objects to construct a set of candidate objects, excluding those objects that are unlikely to correspond to traffic lights. The filtering may be done based on various attributes associated with traffic lights, such as shape, size, texture, position (e.g., relative to the vehicle 200), etc. Such attributes may be based on multiple examples of traffic lights and traffic control signals and stored in a database. In some embodiments, the processing unit 110 may perform a multi-frame analysis on a set of candidate objects that reflect possible traffic lights. For example, the processing unit 110 may track candidate objects across consecutive image frames, estimate the real-world position of the candidate objects, and filter out those objects that are moving (which are unlikely to be traffic lights). In some embodiments, the processing unit 110 may perform a color analysis of the candidate objects and identify the relative positions of the detected colors that appear inside the possible traffic lights.

[0173] At step 562, processing unit 110 may analyze the geometry of the hub. The analysis may be based on any combination of: (i) the number of lanes detected on either side of vehicle 200; (ii) markings detected on the road (such as arrow markings); and (iii) descriptions of the hub extracted from map data (such as data from map database 160). Processing unit 110 may use information obtained from executing monocular analysis module 402 to perform the analysis. In addition, processing unit 110 may determine the correspondence between the traffic lights detected at step 560 and the lanes present near vehicle 200.

[0174] As vehicle 200 approaches a hub, processing unit 110 may, at step 564, update a confidence level associated with the analyzed hub geometry and detected traffic lights. For example, the estimated number of traffic lights present at a hub compared to the number of traffic lights actually present at the hub may affect the confidence level. Thus, based on the confidence level, processing unit 110 may authorize the driver of vehicle 200 to take control in order to improve safety conditions. By executing steps 560-564, processing unit 110 may identify traffic lights present within a set of captured images and analyze hub geometry information. Based on this identification and analysis, processing unit 110 may generate one or more navigation responses for vehicle 200, which are described above in conjunction with Figure 5A describe.

[0175] Figure 5E 5 is a flow chart showing an exemplary process 500E for generating one or more navigation responses in vehicle 200 based on a vehicle path consistent with the disclosed embodiments. At step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. The vehicle path may be represented by a set of points expressed as coordinates (x, z), and the distance d between two points of the set of points may be i May fall within the range of 1 meter to 5 meters. In one embodiment, the processing unit 110 may construct an initial vehicle path using two polynomials, such as left and right road polynomials. The processing unit 110 may calculate the geometric midpoint between the two polynomials and, if there is an offset, offset each point included in the resulting vehicle path by a predetermined offset (e.g., a smart lane offset), (a zero offset may correspond to traveling in the middle of a lane). The direction of the offset may be perpendicular to the segment between any two points in the vehicle path. In another embodiment, the processing unit 110 may use a polynomial and an estimated lane width to offset each point of the vehicle path by half the estimated lane width plus a predetermined offset (e.g., a smart lane offset).

[0176] At step 572, the processing unit 110 may update the vehicle path constructed at step 570. The processing unit 110 may reconstruct the vehicle path constructed at step 570 using a higher resolution so that the distance d between two points of a set of points representing the vehicle path is k Smaller than the distance d described above i For example, the distance d k It may fall within the range of 0.1 m to 0.3 m. The processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm, which may generate a cumulative distance vector S corresponding to the total length of the vehicle path (ie, based on a set of points representing the vehicle path).

[0177] At step 574, the processing unit 110 may determine a look-ahead point (with coordinates (x l , z l ) expression). The processing unit 110 may extract a look-ahead point from the accumulated distance vector S, and the look-ahead point may be associated with a look-ahead distance and a look-ahead time. The look-ahead distance, which may have a lower limit ranging from 10 meters to 20 meters, may be calculated as the product of the speed of the vehicle 200 and the look-ahead time. For example, as the speed of the vehicle 200 decreases, the look-ahead distance may also decrease (e.g., until the look-ahead distance reaches a lower limit). The look-ahead time, which may range from 0.5 seconds to 1.5 seconds, may be inversely proportional to the gain of one or more control loops associated with generating a navigation response in the vehicle 200, such as a heading error of a tracking control loop. For example, the gain of the heading error of the tracking control loop may depend on the bandwidth of the yaw rate loop, the steering actuator loop, the lateral dynamics of the vehicle, etc. Thus, the greater the gain of the heading error of the tracking control loop, the shorter the look-ahead time.

[0178] At step 576, processing unit 110 may determine a heading error and a yaw rate command based on the look-ahead point determined at step 574. Processing unit 110 may calculate the heading error and the yaw rate command by calculating the inverse tangent (e.g., arctan(x l / z l )) to determine the heading error. The processing unit 110 may determine the yaw rate command as the product of the heading error and the high level control gain. If the look-ahead distance is not at the lower limit, the high level control gain may be equal to: (2 / look-ahead time). Otherwise, the high level control gain may be equal to: (2 times the speed of the vehicle 200 / look-ahead distance).

[0179] Fig. 5F 5 is a flow chart showing an exemplary process 500F for determining whether a preceding vehicle has changed lanes consistent with the disclosed embodiments. In step 580, the processing unit 110 may determine navigation information associated with the preceding vehicle (e.g., a vehicle traveling in front of the vehicle 200). For example, the processing unit 110 may use the above combined Figure 5A and Figure 5BThe processing unit 110 may also use the above combined Figure 5E The described techniques determine one or more road polynomials, look-ahead points (associated with vehicle 200 ), and / or slow moving traces (eg, a set of points describing a path taken by a leading vehicle).

[0180] At step 582, processing unit 110 may analyze the navigation information determined at step 580. In one embodiment, processing unit 110 may calculate the distance between the slow-moving trace and the road polynomial (e.g., along the trace). If the variance of this distance along the trace exceeds a predetermined threshold (e.g., 0.1 meters to 0.2 meters on a straight road, 0.3 meters to 0.4 meters on a moderately curved road, and 0.5 meters to 0.6 meters on a sharply curved road), processing unit 110 may determine that the leading vehicle may change lanes. In the event that multiple vehicles are detected traveling ahead of vehicle 200, processing unit 110 may compare the slow-moving traces associated with each vehicle. Based on the comparison, processing unit 110 may determine that a vehicle whose slow-moving trace does not match the slow-moving traces of other vehicles may change lanes. Processing unit 110 may additionally compare the curvature of the slow-moving trace (associated with the leading vehicle) with the expected curvature of the road segment on which the leading vehicle is traveling. The expected curvature may be extracted from map data (e.g., data from the map database 160), road polynomials, slow-moving traces of other vehicles, prior knowledge about the road, etc. If the curvature difference between the slow-moving trace and the expected curvature of the road segment exceeds a predetermined threshold, the processing unit 110 may determine that the preceding vehicle may change lanes.

[0181] In another embodiment, the processing unit 110 may compare the instantaneous position of the preceding vehicle to the look-ahead point (associated with the vehicle 200) over a specific time period (e.g., 0.5 seconds to 1.5 seconds). If the distance between the preceding vehicle's instantaneous position and the look-ahead point changes during the specific time period, and the cumulative sum of the changes exceeds a predetermined threshold (e.g., 0.3 meters to 0.4 meters on a straight road, 0.7 meters to 0.8 meters on a moderately curved road, and 1.3 meters to 1.7 meters on a sharply curved road), the processing unit 110 may determine that the preceding vehicle may change lanes. In another embodiment, the processing unit 110 may analyze the geometry of the slowly moving trail by comparing the lateral distance traveled along the trail to the expected curvature of the slowly moving trail. The expected radius of curvature may be determined according to the following calculation: (δ z 2 +δ x 2 ) / 2 / (δ x ), where δ x represents the lateral distance traveled, and δ zRepresents the longitudinal distance traveled. If the difference between the lateral distance traveled and the expected curvature exceeds a predetermined threshold (e.g., 500 meters to 700 meters), the processing unit 110 may determine that the leading vehicle may change lanes. In another embodiment, the processing unit 110 may analyze the position of the leading vehicle. If the position of the leading vehicle obscures the road polynomial (e.g., the leading vehicle is overlaid on top of the road polynomial), the processing unit 110 may determine that the leading vehicle may change lanes. In the case where the position of the leading vehicle is such that another vehicle is detected ahead of the leading vehicle and the slow-moving tracks of the two vehicles are not parallel, the processing unit 110 may determine that the (closer) leading vehicle may change lanes.

[0182] At step 584, processing unit 110 may determine whether leading vehicle 200 has changed lanes based on the analysis performed at step 582. For example, processing unit 110 may make a determination based on a weighted average of the various analyses performed at step 582. Under such a scheme, for example, a determination by processing unit 110 that a leading vehicle may change lanes based on a particular type of analysis may be assigned a value of "1" (whereas "0" indicates a determination that a leading vehicle may not change lanes). Different analyses performed at step 582 may be assigned different weights, and the disclosed embodiments are not limited to any particular combination of analyses and weights.

[0183] Figure 6 6 is a flow chart illustrating an exemplary process 600 for generating one or more navigation responses based on stereo image analysis consistent with disclosed embodiments. At step 610, processing unit 110 may receive first and second pluralities of images via data interface 128. For example, a camera included in image acquisition unit 120 (such as image capture devices 122 and 124 having fields of view 202 and 204) may capture first and second pluralities of images of an area in front of vehicle 200 and send the first and second pluralities of images to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first and second pluralities of images via two or more data interfaces. Disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0184] At step 620, the processing unit 110 may execute the stereo image analysis module 404 to perform stereo image analysis of the first and second plurality of images to create a three-dimensional map of the road ahead of the vehicle and detect features within the image, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. The stereo image analysis may be performed in a manner similar to that described above in conjunction with FIG. 5A to FIG. 5D- For example, the processing unit 110 may execute the stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road signs, traffic lights, road hazards, etc.) within the first and second multiple images, filter out a subset of the candidate objects based on various criteria, and perform multi-frame analysis, construct measurements, and determine confidence levels for the remaining candidate objects. In performing the above steps, the processing unit 110 may consider information from both the first and second multiple images, rather than information from a single set of images. For example, the processing unit 110 may analyze differences in pixel-level data (or other subsets of data from the two captured image streams) for candidate objects that appear in both the first and second multiple images. As another example, the processing unit 110 may estimate the position and / or velocity of the candidate object (e.g., relative to the vehicle 200) by observing that the object appears in one of the multiple images but not in the other, or other differences that may exist relative to the objects that appear in the two image streams. For example, the position, velocity, and / or acceleration relative to the vehicle 200 may be determined based on the trajectory, position, movement characteristics, etc. of features associated with the object that appear in one or both of the image streams.

[0185] In step 630, the processing unit 110 may perform the following operations based on the analysis performed in step 620 and the above combination. Figure 4 The techniques described above may be used to execute the navigation response module 408 to generate one or more navigation responses in the vehicle 200. For example, the navigation response may include a turn, a lane shift, an acceleration change, a speed change, braking, etc. In some embodiments, the processing unit 110 may use data obtained from executing the speed and acceleration module 406 to generate one or more navigation responses. In addition, multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof.

[0186] Figure 7is a flow chart showing an exemplary process 700 for generating one or more navigation responses based on analysis of three sets of images consistent with the disclosed embodiments. At step 710, the processing unit 110 may receive the first, second, and third pluralities of images via the data interface 128. For example, a camera included in the image acquisition unit 120 (such as the image capture devices 122, 124, and 126 having the fields of view 202, 204, and 206) may capture the first, second, and third pluralities of images of the area in front of and / or to the sides of the vehicle 200 and send the first, second, and third pluralities of images to the processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, the processing unit 110 may receive the first, second, and third pluralities of images via more than three data interfaces. For example, each of the image capture devices 122, 124, 126 may have an associated data interface for communicating data with the processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0187] In step 720, processing unit 110 may analyze the first, second, and third plurality of images to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. This analysis may be performed similarly to the above combined FIG. 5A to FIG. 5D and Figure 6 For example, the processing unit 110 may perform a monocular image analysis on each of the first, second, and third plurality of images (e.g., by executing the monocular image analysis module 402 and based on the above combined FIG. 5A to FIG. 5D Alternatively, the processing unit 110 may perform stereoscopic image analysis on the first and second plurality of images, the second and third plurality of images, and / or the first and third plurality of images (e.g., by executing the stereoscopic image analysis module 404 and based on the above combined Figure 6 4, and / or 5. The processing unit 110 may perform a combination of monocular image analysis and stereoscopic image analysis. For example, the processing unit 110 may perform monocular image analysis on the first plurality of images (e.g., via execution of the monocular image analysis module 402) and stereoscopic image analysis on the second and third plurality of images (e.g., via execution of the stereoscopic image analysis module 404). The configuration of the image capture devices 122, 124, and 126 (including their respective locations and fields of view 202, 204, and 206) may affect the type of analysis performed on the first, second, and third plurality of images. The disclosed embodiments are not limited to a particular configuration of image capture devices 122, 124, and 126, or the type of analysis performed on the first, second, and third plurality of images.

[0188] In some embodiments, the processing unit 110 may perform testing of the system 100 based on the images acquired and analyzed in steps 710 and 720. Such testing may provide an indicator of the overall performance of the system 100 for certain configurations of the image capture devices 122, 124, and 126. For example, the processing unit 110 may determine the ratio of "false hits" (e.g., situations where the system 100 incorrectly determines that a vehicle or pedestrian is present) to "misses."

[0189] At step 730, the processing unit 110 may generate one or more navigation responses in the vehicle 200 based on information obtained from both of the first, second, and third plurality of images. The selection of both of the first, second, and third plurality of images may depend on various factors, such as the number, type, and size of objects detected in each of the plurality of images. The processing unit 110 may also make the selection based on the following: image quality and resolution, effective field of view reflected in the image, the number of captured frames, the degree to which one or more objects of interest actually appear in the frame (e.g., the percentage of frames in which the object appears, the proportion of the object that appears in each such frame, etc.), etc.

[0190] In some embodiments, the processing unit 110 may select information obtained from both of the first, second, and third pluralities of images by determining how consistent the information obtained from one image source is with information obtained from other image sources. For example, the processing unit 110 may combine processed information obtained from each of the image capture devices 122, 124, and 126 (whether through monocular analysis, stereo analysis, or any combination of both) and determine that consistent visual indicators (e.g., lane markings, detected vehicles and their locations and / or paths, detected traffic lights, etc.) are presented across the images captured from each of the image capture devices 122, 124, and 126. The processing unit 110 may also exclude information that is inconsistent across the captured images (e.g., vehicles changing lanes, lane models indicating vehicles that are too close to the vehicle 200, etc.). Thus, the processing unit 110 may select information obtained from both of the first, second, and third pluralities of images based on the determination of consistent and inconsistent information.

[0191] For example, the navigation response may include a turn, a lane shift, an acceleration change, etc. The processing unit 110 may be based on the analysis performed in step 720 and the above combined Figure 4 The processing unit 110 may also use data obtained from executing the speed and acceleration module 406 to generate one or more navigation responses. In some embodiments, the processing unit 110 may generate one or more navigation responses based on the relative position, relative velocity, and / or relative acceleration between the vehicle 200 and the detected objects within any of the first, second, and third plurality of images. The multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof.

[0192] Lane end recognition system

[0193] The system 100 can identify when a driving lane will terminate and, when the termination is detected, make responsive navigation changes in the vehicle 200's journey, such as lane changes. A lane may refer to a designated or intended path of travel for a vehicle and may have marked boundaries (e.g., lines on a road) or unmarked boundaries (e.g., road curbs, road barriers, guardrails, parked vehicles, etc.) or constraints. The system can operate based solely on visual information, rather than relying on map data for lane end determination. However, in some cases, the system may rely on a combination of map data and visual information relative to the lane of travel. Further, while the system can identify the end of a lane by direct observation of the lane termination, the system can also identify the approaching lane end earlier by, for example, observing and recognizing road sign warnings that provide information about the lane termination.

[0194] Figure 8 Vehicle 200 is illustrated traveling on road 800 in which the disclosed systems and methods for identifying the end of a lane of travel and operating vehicle 200 may be used. Although vehicle 200 is described as being equipped with image capture devices 122 and 124, more or fewer cameras may be employed on any particular vehicle 200. As shown, road 800 may be subdivided into lanes, such as lanes 810 and 820. Lanes 810 and 820 are shown as examples; a given road 800 may have additional lanes based on the size and nature of the road, such as an interstate highway. Figure 8 In the example of , vehicle 200 is traveling in lane 810 and can see that lane 810 will soon end. In response, system 100 can cause vehicle 200 to turn to lane 820 as discussed in detail below.

[0195] The processing unit 110 may be configured to determine one or more lane constraints associated with each of the lanes 810 and 820 based on a plurality of images acquired by the image capture devices 122-126 that the processing unit 110 may receive via the data interface 128. According to some embodiments, the lane constraints may be identified by visible lane boundaries (such as dashed or solid lines marked on the road surface). Additionally or alternatively, the lane constraints may include a roadside or obstacle on the road surface. Additionally or alternatively, the lane constraints may include markings (e.g., Botts points). According to some embodiments, the processing unit 110 (via the lane constraint module 910 and / or the lane positioning module 920 described in detail below) may determine the constraints associated with the lanes 810 / 820 by identifying the midpoint of the road width of the entirety of the road 800 or the midpoint of one of the lanes 810 / 820. When, for example, a line designating a road lane such as the lane 810 / 820 is not drawn or marked, the processing unit 110 may identify the lane constraints by alternative means (such as by estimation or inference) based on known road parameters.

[0196] Detection of the road 800 and the constraints and paths that make up the lanes 810 / 820 may include the processing unit 110 determining a 3D model thereof via a camera coordinate system. For example, the 3D model of the lanes 810 / 820 may be described by a cubic polynomial. In addition to the 3D modeling of the driving lanes, the processing unit 110 may also perform multi-frame estimation of host motion parameters such as the speed, yaw and pitch rates, and acceleration of the vehicle 200. The processing unit 110 may receive information pertaining to these parameters from, for example, a speed sensor, a multi-axis accelerometer, etc. included in the vehicle 200. Optionally, the processing unit 110 may detect stationary and moving vehicles and their positions, headings, velocities, and accelerations (all relative to the vehicle 200, which will be discussed below in conjunction with the present invention). Fig. 10B The processing unit 110 may further determine a road elevation model to convert all information obtained from the plurality of images into a three-dimensional space.

[0197] Generally, as a default condition, the vehicle 200 may be relatively centered within the lane 810 / 820 so that the distance from the vehicle 200 to the lane 810 / 820 is relatively small. Figure 8 810 / 820 are equal or nearly equal. However, in some situations, environmental factors may make this undesirable or impractical, such as when an object or structure exists on one side of the road. Thus, there may be situations where it may be desirable or advantageous for the vehicle 200 to be closer to one lane constraint or the other (i.e., x1>x2 or vice versa), particularly if a lane is determined to be about to end, as will be discussed below in conjunction with Fig. 10A , Fig. 10B , Fig.11for discussion.

[0198] Fig. 9 1 is an exemplary block diagram of memory 140 and / or 150 consistent with the disclosed embodiments, which memory 140 and / or 150 may store instructions for performing one or more operations. Fig. 9 As illustrated in , the memory 140 may store one or more modules for performing the lane end recognition and response described herein. For example, the memory 140 may store a lane constraint module 910, a lane positioning module 920, and an action module 930. The application processor 180 and / or the image processor 190 may execute instructions stored in any module 910-930 (included in the memory 140). It will be understood by those skilled in the art that references to the processing unit 110 discussed below may refer to the application processor 180 and the image processor 190 individually or collectively. Therefore, any of the following processing steps may be performed by one or more processing devices.

[0199] The lane constraint module 910 may store instructions that, when executed by the processing unit 110, may detect and define boundaries associated with lanes such as lanes 810 and 820. In some embodiments, the detected boundaries may be artificial, such as painted lines or reflective objects. In other embodiments, the detected boundaries may simply include the natural curvature of the road or the end of a lane or road surface. The lane offset module 910 may process multiple images received from at least one image capture device 122-124 to detect lane constraints. As discussed above, this may include identifying painted lane lines and / or measuring the midpoint of the road surface. In addition, the lane constraint module 910 may detect the route and / or constraints of the driving lanes such as lanes 810 and 820 by extracting information from one or more road signs within the field of view of the image capture devices 122-124. Examples of these road signs may include, without limitation, signs containing text messages such as "Lane Ends - Merge Left" or signs containing visual messages such as visual representations of lane termination.

[0200] Lane localization module 920 may store instructions that, when executed by processing unit 110, may assist system 100 in determining the location of vehicle 200. In some embodiments, the step of determining the location of vehicle 200 may include determining the distance from the vehicle to one or more lane constraints associated with the current lane in which the vehicle is traveling based on at least one indicator of the location of the vehicle. This determination of the location of vehicle 200 may be based at least in part on an analysis of image data captured by one or more image capture devices 122-126. In some embodiments, the image data may be supplemented or replaced with map data (e.g., GPS information) and / or data from other sensors (e.g., laser range finders, etc.). In such embodiments, the distance relative to the lane constraints identified by lane constraint module 910 may be measured. For example, the distance to one or more lane constraints may specify the distance to the end of a lane or an area where a lane has partially terminated (e.g., for a lane including a merge area that reduces the distance until the lane has terminated). In order to determine the location of vehicle 200, lane localization module 920 may process multiple images to detect features of the road on which vehicle 200 is traveling. Additionally or alternatively, the lane offset module 920 may receive information indicating other features present in the image data (such as other vehicles, curvature of the road, etc.) from other modules (including the position sensor 130) or other systems. The lane positioning module 920 may periodically update the position state of the vehicle using a coordinate system or other position representation (e.g., map data from a mapping application, GPS information, etc.). The state may be stored within the system, such as within the memory 140 / 150. The lane positioning module 920 may update the position state while the vehicle 200 moves at set time points (such as every minute, after a predetermined number of minutes, every hour, etc.). Alternatively, the lane positioning module 920 may update the position state upon detecting a specific vehicle operation (such as changing lanes, accelerating, decelerating, or other steering operations).

[0201] The action module 930 may store instructions that, when executed by the processing unit 110, may assist the system 100 in performing one or more actions relative to the operation of the vehicle 200 based on information received from one or more sources, such as the position sensor 130, the lane constraint module 910, or the lane positioning module 920. In some embodiments, the action module 930 may receive information indicating that the current lane of travel of the vehicle 200 ends. The action module 930 may then perform one or more actions that affect the operating state of the vehicle 200, such as causing the system 100 to provide control signals to one or more of the throttle system 220, the brake system 230, and the steering system 240 to navigate the vehicle 200 (e.g., by causing acceleration, turning, lane shifting, etc.). For example, in some embodiments, the action module 930 may send an instruction to the steering system 240, and the steering system 240 may execute the instruction to steer the vehicle into a new lane of travel. Additionally or alternatively, the action module 300 can send instructions to other systems associated with the vehicle 200, such as the brake system 230, the turn signal, the throttle system 220, etc. In some embodiments, the action module 930 can instead provide a human operator of the vehicle with audio, visual, tactile, or haptic feedback representative of information collected from associated systems and / or sensors. The human operator can then act on the feedback to steer the vehicle into a different lane.

[0202] Fig. 10A Provided Figure 8 . Vehicle 200 is again traveling in lane 810 of road 800, which is about to end. Vehicle 200 is again equipped with image capture devices 122 and 124; more or fewer devices may be associated with any particular vehicle 200.

[0203] To the right of road 800 are road signs 1002 and 1004. These signs are exemplary and not intended to be limiting. Road sign 1002 presents a visual depiction of a lane termination. This sign is familiar to many drivers and indicates the need to merge. Sign 1004 is typically placed closer to the actual end of the lane, and this sign 1004 informs the driver "LANE ENDS - MERGE LEFT". Consistent with the disclosed embodiments, system 100 can be configured to extract information from road signs such as signs 1002 and 1004 to determine whether and when the current travel lane in which vehicle 200 is traveling terminates. For visual signs such as sign 1002, image capture devices 122-126 can capture an image of the entire face of the sign. Image processor 190 can be configured to recognize an image graphic of sign 1002 and can send information indicating that the sign indicates a lane termination to other modules or units within system 100. In some embodiments, a constituent module or subsystem of the system 100 (such as memory 140 / 150) may be configured to contain a list or database of typical road signs and patterns for rapid, real-time detection and information extraction purposes. The sign 1004 contains only text, so its warning can be read by the system 100 through optical character recognition (OCR) capabilities, which may be configured within the processing unit 110, memory 140 / 150, image processor 190, or any other constituent subsystem or module of the system 100.

[0204] Extracting information from road markings is one of a number of actions that system 100 may be configured to perform for lane recognition purposes. Additionally, via lane constraint module 910 and lane positioning module 920 within memory 140 / 150, system 100 may be configured to perform visual determination of road 800 and accompanying distance measurements based on at least one indicator of the vehicle's position. Fig. 10A In the example of FIG. 1 , various potential measurements that can be used by system 100 are shown. System 100 can obtain measurements by, for example, analyzing one or more images obtained by one or more image capture devices 122-126. Measurement 1006 can represent the total width w of lane 810. Distance w can also be recorded as the distance from Figure 8The sum of the distances x1 and x2 and the width of the vehicle 200. Measurement 1008 may represent the radius of curvature r of the road 800 as the lane 810 terminates. A shorter radius r may indicate that the system 100 has a greater urgency to steer the vehicle 200 into the lane 820. Measurement 1010 may represent the distance d1 from the vehicle 200 to a constraint associated with the lane 810. The constraint may indicate that the lane 810 terminates. As discussed above, the constraint may be a drawn line, or may be a natural edge of the road 800. The distance d1 may also be measured to any portion in three-dimensional space of an obstacle, such as a guardrail, wall, barrel, cone, or any other such object. Distance d1 and other such distances (such as x1, x2, and w) may be measured from any portion of the interior or exterior vehicle 200, including, but not limited to: the front of the vehicle 200, portions of the vehicle 200 such as headlights or front license plates, mounted locations of image capture devices 122 or 124, a determined centroid of the vehicle 200, the rear of the vehicle 200, one or more windshields or reflectors associated with the vehicle 200, wheels of the vehicle 200, the right or left side or windows of the vehicle 200, points associated with the roof of the vehicle 200, or points associated with the chassis of the vehicle 200. As will be described below in conjunction with Fig.11 As discussed in the process 1100 , the system 100 may use the extracted road sign information or one or both of the distances 1006 - 1010 to determine whether the current lane of travel of the vehicle 200 is about to terminate.

[0205] Fig. 10B exemplifies a set of additional visual determinations and / or distance measurements that the system 100 may use to determine whether the current lane of travel terminates on a road. Fig. 10B In the example of , there are two vehicles 200a and 200b traveling in lane 810 of road 800 that is about to end. In some embodiments, vehicle 200a can be equipped with system 100 and can measure distance d2 to the rear side of vehicle 200b. Distance d2 can be measured to or from any part of vehicles 200a and 200b discussed previously. Alternatively, vehicle 200b can be equipped with system 100 and can measure distance d2 to the rear to vehicle 200a. In order Fig. 10B In the exemplary illustration, both vehicles are described as being equipped with the system 100. In some embodiments, the system 100 (in Fig. 10B) can be configured to make a visual determination of whether the current lane of travel terminates based on the distance d2. In these embodiments, the system 100 can be configured to identify the lane termination based on a sudden shortening of the distance d2. Alternatively, the system 100 can detect that the distance d2 has acquired a lateral element, rather than as a straight line extending perpendicular to the front plane associated with the vehicle 200a. When configured in this manner, measurements such as c1-c4 can be measured and used for position state determination by the lane positioning module 920. The distance c1-c4 can represent the distance from one side of the vehicle 200a or 200b to the edge of the lane 810, whether it is the lane constraint 1014 or the center imaginary line in the middle of the road 800 dividing the lane 810 / 820. In other embodiments, the distance to the lane constraint 1012 can be measured on the far side of the lane 820 (not shown).

[0206] like Figure 8 and Fig. 10A middle, Fig. 10B Lane 810 in is about to end, and both vehicles 200a and 200b will need to merge into lane 820. Fig. 10B 1 illustrates additional distances that the system 100 may be configured to calculate, extract, and / or analyze in a live driving situation. In addition to the distances c1-c4 described above, the system 100 may also be configured to calculate the distances w1 and w2 of the lane 810 relative to one or more lane constraints associated with the lane and the midpoint m. When added together, Fig. 10A As shown in , distances w1 and w2 are equal to measurement 1006w. System 100 can periodically measure distances w1 and w2 and midpoint m of lane 810 at set time points via lane positioning module 920. If distances w1 and w2 decrease, and / or if midpoint m shifts to the right or left, system 100 can determine that lane 810 is about to end and needs to merge into another lane.

[0207] Fig.11 A process 1100 for detecting a lane termination for a vehicle consistent with disclosed embodiments is illustrated. The steps of process 1100 may be performed by one or more of processing unit 110, image acquisition unit 120, position sensor 130, lane constraint module 910, lane positioning module 920, or action module 930.

[0208] At step 1110, process 1100 may include acquiring at least one image of an area in front of vehicle 200 using at least one image capture device 122, 124, and / or 126. The area may also include at least one road sign (such as road signs 1002 and / or 1004) that provides information indicating the end of a lane. Processing unit 110 may receive a plurality of images from the image capture device via data interface 128. The at least one image may then be processed in real time by image processor 190 of processing unit 110.

[0209] At step 1120, the process 1100 may extract lane termination information from a representation of at least one road sign included in at least one image. The extraction may be performed by one or more of the image processor 190, the lane constraint module 910, or the lane positioning module 920. Various lane termination information may be extracted from the image data. For example, on a road such as road 800 that includes multiple driving lanes (such as lanes 810 and 820), the extracted lane termination information may include identifying which of the multiple lanes is about to terminate. In other embodiments, the extracted lane termination information may include the distance to the lane termination. The image processor 190, the lane constraint module 910, and / or the lane positioning module 920 may identify one or more objects or lane constraints to help determine the distance to the lane termination. Fig. 10AFor example, one or more images received from the data interface 128 may include road signs 1002 and 1004, and may be factored into the road curvature 1008 to determine the distance 1010 remaining in the lane. In some embodiments, the value of the curvature 1008 itself may independently constitute the extracted lane termination information. As part of processing the extracted information, the system 100 may use the position sensor 130 to determine the distance from the vehicle 200 to one or more of the identified objects (e.g., lane constraints). In some embodiments, the system 100 may determine the distance from the vehicle 200 to one or more of the identified objects (e.g., lane constraints) by analyzing one or more images acquired by one or more image capture devices 122-126. In other embodiments, the system 100 may determine the distance from the vehicle 200 to one or more identified objects (e.g., lane constraints) by using a combination of data provided by the position sensor 130 and data obtained by analyzing one or more images acquired by one or more image capture devices 122-126. At step 1130, the process 1100 may determine the location of a vehicle (such as vehicle 200) based on at least one indicator of the location of the vehicle. The at least one indicator of the location of the vehicle may include information obtained by a direct measurement tool, physical device or module (such as position sensor 130). Alternatively or additionally, the at least one indicator of the location of the vehicle may include information obtained by indirect measurement. For example, in some embodiments, the at least one indicator of the location may include a visual determination of which lane (such as lanes 810 and 820) the vehicle 200 may be traveling on among a plurality of lanes on the road 800. In other embodiments, the at least one indicator of the location may include one or more visual determinations. The visual determination may be determined directly by software instructions executed by the image acquisition unit 120 in a "rangefinder" manner, or may be determined from an analysis of image data received by the image processor 190 from the data interface 128 and acquired by one or more image capture devices 122-126. In some embodiments, the at least one indicator of the location may include a visual determination of the distance to the end of the lane. This determination can be made, for example, by lane constraint module 910 and lane positioning module 920 based on a determined distance from vehicle 200 to one or more lane constraints of road 800 associated with one or more lanes. In other embodiments, at least one indicator of location can include a visual determination of the distance to a second vehicle. The second vehicle can be traveling in front of vehicle 200. This is in conjunction with the above Fig. 10B Once determined, the location information may be stored within the system 100, such as in the memory 140 / 150, and may be combined with the above Fig. 9The discussed updates are performed periodically by the lane positioning module 920 .

[0210] At step 1140, the process 1100 may determine (based on the extracted lane information from step 1120 and the determined distance from the vehicle to one or more lane constraints of the road 800 from step 1130) whether the current lane in which the vehicle is traveling is about to end. As generally described above, the system 100 may determine the ongoing state of lanes of travel within the road 800, such as lanes 810 and 820. For example, the system 100 may extract information from road signs in front of the vehicle, such as road signs 1002 and 1004 indicating that the lane in which the vehicle 200 is traveling may soon end. In addition, various elements of the system 100 may be used to make distance measurements. For example, the position sensor 130 and / or the lane positioning module 920 may send information about the relative position of the vehicle 200 within the lane 810 / 810, including how far the vehicle 200 is from the end of the lane, to the processing unit 110. The end of the lane may be determined based on a measured distance or visually (eg, by analyzing one or more images acquired by one or more image capture devices 122 - 126 ).

[0211] If the processing unit 110 determines from the acquired information that the current lane of travel does not terminate (step 1140: No), the process 1100 can return to the beginning and the system 100 can continue to monitor the status of the vehicle 200 relative to its current lane of travel. For example, if a road sign indicates that the lane is about to terminate due to road construction, however, due to major exceptions (such as weather, workday, or other factors), there are actually no workers or obstacles blocking the lane. The system 100 can also determine, for example via the lane constraint module 910, that a perceived deviation in one or more lane constraints may be due to a new or temporary road pattern, or may include natural curvature in the road. Of course, the driver of the vehicle 200 (if any) is still able to change lanes manually.

[0212] Alternatively, if the system 100 determines that the current lane of travel is about to terminate (step 1140: yes), the process 1100 can proceed to step 1150. At step 1150, the process 1100 can cause the vehicle 200 to change lanes to another lane of the road 800 that does not terminate. The processing unit 110 can send instructions to one or more systems associated with the vehicle 200 via the action module 930. For example, instructions can be sent to the steering system 240, instructing the vehicle 200 to merge into the next lane to the left or right. Other instructions can be sent; for example, the image processor 190 can determine from the real-time image data that traffic prevents merging into the next lane immediately. In these embodiments, instructions can first be sent to the braking system 230 and / or the turn signal control system, and the vehicle 200 can be slowed down or stopped until the desired new lane is safely merged. At this time, instructions to the steering system 220 can be made and the vehicle can turn. In some alternative embodiments, where the vehicle 200 is partially or fully under the control of a human operator, step 1150 may instead include generating feedback to the operator notifying that the lane of travel is about to end and a lane change is required. In some embodiments, if the human operator does not respond to the feedback within a certain period of time, an automatic system such as system 100 and its action module 930 may take over the operation of the vehicle and cause the vehicle to automatically change lanes as described above. The time period may be predetermined or may be calculated based on a visual determination of the distance to the end of the lane, such as the visual determination described above in conjunction with step 1130.

[0213] Detection of low-height objects on the road

[0214] The system 100 can detect when an object is within the current lane of travel, and if the system 100 determines that the object exceeds a threshold height, the system can steer the associated vehicle around or over the object. A "lane" can refer to a designated or intended path of travel for a vehicle and can have marked boundaries (e.g., lines on a road) or unmarked boundaries (e.g., road curbs, road barriers, guardrails, parked vehicles, etc.) or constraints. The system 100 can operate solely based on visual information acquired by one or more image acquisition units to make these detections and determinations. In some embodiments, the system 100 can be configured to detect objects on a road that protrude only 2 cm above the road surface.

[0215] Fig.12Vehicle 200 is illustrated traveling on road 800 in which the disclosed systems and methods for detecting objects on the road may be used. Vehicle 200 is depicted as being equipped with image capture devices 122 and 124; more or fewer image capture devices (e.g., including cameras) may be employed on any particular vehicle 200. As shown, road 800 may be subdivided into lanes, such as lanes 810 and 820. Lanes 810 and 820 are shown as examples; a given road 800 may have additional lanes based on the size and nature of the road, such as an interstate highway. Fig.12 In the example of FIG. 8 , vehicle 200 is traveling in lane 810, and two objects 1202 and 1204 are located in lane 810. As discussed in detail below, system 100 can cause vehicle 200 to steer around or over object(s).

[0216] The processing unit 110 may be configured to determine one or more lane constraints associated with each of the lanes 810 and 820 based on a plurality of images acquired by the image capture devices 122-126 that the processing unit 110 may receive via the data interface 128. According to some embodiments, the lane constraints may be identified by visible lane boundaries (such as dashed or solid lines marked on the road surface). Additionally or alternatively, the lane constraints may include a curb or obstacle of the road surface. Additionally or alternatively, the lane constraints may include markings (e.g., Botts points). According to some embodiments, the processing unit 110 (via the lane positioning module 1320 described in detail below) may determine the constraints associated with the lanes 810 / 820 by identifying the midpoint of the road width of the entirety of the road 800 or the midpoint of one of the lanes 810 / 820. When, for example, a line designating a road lane such as the lane 810 / 820 is not drawn or marked, the processing unit 110 may identify the lane constraints by alternative means (such as by estimation or inference) based on known road parameters.

[0217] The detection of the road 800 and the constraints and paths constituting the lanes 810 / 820 may include the processing unit 110 determining a 3D model thereof via the camera coordinate system. For example, the 3D model of the lane 810 / 820 may be described by a cubic polynomial. In addition to the 3D modeling of the driving lane, the processing unit 110 may also perform multi-frame estimation of host motion parameters such as the speed, yaw and pitch rates and / or acceleration of the vehicle 200. The processing unit 110 may detect stationary and moving objects and their positions relative to the vehicle 200, which will be discussed below in conjunction with Fig.14A , Fig. 14B and Fig.15For example, the processing unit 110 may use image data, GPS data, and / or data from sensors (e.g., position sensor 130, speed sensor, etc.) to determine the position of the vehicle 200. The processing unit 110 may further determine a road elevation model to convert part or all of the information obtained from the multiple images into a three-dimensional space.

[0218] exist Fig.12 In the example of , multiple measurements represent what the system 100 may consider when determining potential actions to be taken using the action module 1330, which will be described in detail below. Fig.12 , represented by a cardboard box and a ladder, respectively. These are common items that would be seen as obstacles on a typical road, for example, after falling from an unseen second vehicle. These objects are used as examples only. The system 100 can be configured to detect the presence of any object that can be found on a road, including, but not limited to, branches, tires or parts thereof, scrap metal or wood, speed bumps inside a travel lane, a human or animal, a pipe, another vehicle, etc.

[0219] As part of a driver assistance object detection system consistent with disclosed embodiments, system 100 may determine, for example, distances l1 and l2, which may represent distances from the surface of vehicle 200 to various features of road 800 and / or lanes 810 / 820 (such as lane constraints, road signs, or any other such features). Further, system 100 may determine distances d3 and d4, which are measured distances from the surface of vehicle 200 to each of objects 1202 and 1204, respectively. Distances such as l1, l2, d3, and d4 may be measured from any portion of the interior or exterior vehicle 200, including, but not limited to: the front of the vehicle 200, portions of the vehicle 200 such as headlights or a front license plate, a mounted location of an image capture device 122 or 124, a determined centroid of the vehicle 200, the rear of the vehicle 200, one or more windshields or reflectors associated with the vehicle 200, wheels of the vehicle 200, the right or left side or windows of the vehicle 200, points associated with the roof of the vehicle 200, or points associated with the chassis of the vehicle 200. Further details of how these exemplary distance measurements may be used to determine potential actions to be taken by the system 100 will be discussed in detail below.

[0220] Fig.13 1 is an exemplary block diagram of memory 140 and / or 150 consistent with the disclosed embodiments, which memory 140 and / or 150 may store instructions for performing one or more operations. Fig.13As illustrated in , memory 140 may store one or more modules for performing object detection and response described herein. For example, memory 140 may store object detection module 1310, lane positioning module 1320, and action module 1330. Application processor 180 and / or image processor 190 may execute instructions stored in any module 1310-1330 (included in memory 140). Those skilled in the art will appreciate that references to processing unit 110 discussed below may refer to application processor 180 and image processor 190 individually or collectively. Therefore, any of the following processing steps may be performed by one or more processing devices.

[0221] Object detection module 1310 may store instructions that, when executed by processing unit 110, may detect one or more objects found within road 800 (such as within lanes 810 and 820). Fig.14A , Fig. 14B and Fig.15 As discussed, the object detection module 1310 can identify one or more objects within the image data captured by one or more image capture devices 122-126 and determine that the objects are resting on the road surface. The object detection module 1310 can also determine a reference plane corresponding to the road surface, and can use the reference plane to determine the size and attributes of the identified object (such as its height, length, and width). The object detection process can also include identifying drawn lane lines and / or measuring the midpoint of a road surface such as road 800 in order to determine the reference plane and in order to determine the location of objects within the road / lane. In some embodiments, the field of view of the image capture device used to generate the image data can range from about 35 degrees to about 55 degrees.

[0222] Lane localization module 1320 may store instructions that, when executed by processing unit 110, may assist system 100 in determining the location of vehicle 200. In some embodiments, determining the location of vehicle 200 may include determining a distance from the vehicle to one or more lane constraints associated with a current lane in which the vehicle is traveling based on at least one indicator of the vehicle's location. In such embodiments, distances relative to identified objects detected by object detection module 1310 may also be measured. Consistent with disclosed embodiments, distances measured for one or more lane constraints may be used to determine whether a change in the directional route of vehicle 200 is required or recommended. To determine the location of vehicle 200, lane localization module 1320 may process multiple images to detect features of the road on which vehicle 200 is traveling. Additionally or alternatively, lane localization module 1320 may receive information indicating the presence of other features in the image data (such as other vehicles, curvature of the road, etc.) from other modules (including position sensor 130) or other systems.

[0223] The action module 1330 may store instructions that, when executed by the processing unit 110, may assist the system 100 in performing one or more actions relative to the operation of the vehicle 200 based on information received from one or more sources (such as the position sensor 130, the object detection module 1310, or the lane positioning module 1320). In some embodiments, the action module 1330 may receive information (from, for example, the object detection module 1310) indicating that one or more objects on the road 800 have been detected. The information may also include one or more dimensions or attributes of the detected object, such as its height relative to the road surface. The action module 1330 may then perform one or more actions that affect the operating state of the vehicle 200, such as causing the system 100 to provide control signals to one or more of the throttle system 220, the braking system 230, and the steering system 240 to navigate the vehicle 200 (e.g., by causing acceleration, turning, steering, lane shifting, etc.). For example, in some embodiments, the action module 1330 can send instructions to the steering system 240, and the steering system 240 can execute the instructions to steer the vehicle into a new lane of travel. In other embodiments, the action module 1330 can determine that a safer and / or better course of action would be to simply steer the vehicle 200 so that it passes over the top of the object, assuming that the height of the object is lower than the road clearance height of the vehicle 200. Additionally or alternatively, the action module 1330 can send instructions to other systems associated with the vehicle 200 (such as the braking system 230, turn signals, throttle system 220, etc.). In some embodiments, the action module 1330 can instead provide audio, visual, or tactile feedback to the human operator of the vehicle that represents the information collected from the associated systems and / or sensors. The human operator can then act on the feedback to steer the vehicle into a different lane.

[0224] Fig.14A An example of two images among a plurality of captured images that may be acquired by the system 100 via the image capture devices 122-126 is illustrated. Fig.14A , image 1402 (top) may be a first image acquired, and image 1404 (bottom) may be a second image taken later (such as a few seconds later). As can be seen from images 1402 and 1404, a vehicle (such as vehicle 200 above) equipped with system 100 is traveling on a road (such as road 800). The road is aligned directly in front of the vehicle, and the line of sight of the road terminates at ground level 1406. Based on the line of sight illustrated in image 1402, the road may form a slope. Any road construction or alignment can be analyzed by system 100. As can be seen from images 1402 and 1404, object 1202 representing a cardboard box of initially unknown size is located in the middle of the right travel lane of the illustrated road.

[0225] In some embodiments, the system 100 can be configured to analyze the captured images (such as images 1402 and 1404) to determine a reference plane corresponding to a road plane (such as a road surface). Fig.15 The process 1500 is described in more detail below. In brief, the system 100 (via the processing unit 110 and the image processor 190) can determine a reference plane corresponding to a road surface as an average surface level associated with the road surface. This may result in changes in the road surface due to changes in the composition of the road (i.e., asphalt, concrete, or even gravel) or the inclination and / or curvature of the road. Multiple images from among the multiple images can be used to determine the reference plane. In Fig.14A In the example of , the determined reference plane is represented by reference plane 1408.

[0226] System 100 can locate one or more objects (such as object 1202) as being located on the road via processing unit 110 and object detection module 1310. Detection can occur, for example, when an object appears in the same location in multiple images among the multiple images captured. Objects (such as object 1202) can be moving or stationary. Fig.14A In the example of , object 1202 is stationary on the road. The location(s) of the object(s) may be determined by the object detection module 1310 through various methods. In some embodiments, the location(s) of the target object(s) may be determined through GPS and / or other coordinate-based systems. Further, the object detection module 1310 may perform a visual determination of the location of the object 1202. In some embodiments, one or more of the object detection module 1310, the position sensor 130, and the image processor 190 may also determine how far the object 1202 is from the vehicle 200, as described above in conjunction with Fig.12 When configured in this way, it is possible to measure and use Fig.12 Measurements such as d3-d4 shown in are used for position state determination performed by lane positioning module 1320.

[0227] Can be combined FIG. 5A to FIG. 5F , Figure 6 , Figure 7 The distance from the vehicle 200 to an object detected inside the road, such as the object 1202, is calculated in the manner described above. For example, the system 100 may analyze image data, location data (e.g., GPS location information), map data, speed data, and / or data from sensors included in the vehicle 200 to calculate the distance. The system 100 may collect data from, for example, the image acquisition unit 120, the location sensor 130, and other sensors for analysis.

[0228] In some embodiments, the processing unit 110 may construct the measurements based on estimation techniques using a series of time-based observations (such as a Kalman filter or linear quadratic estimation (LQE)) and / or based on available modeling data for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). The Kalman filter may be based on measurements of the size of the object, where the size measurement is proportional to the time to collision (e.g., the amount of time required for the vehicle 200 to reach the object).

[0229] The processing unit 110 may be configured in some embodiments to create a 3D reconstruction of the visible portion of the road using one or more mathematical algorithms or models discussed previously. FIG. 5A to FIG. 5F As discussed, the processing unit 110 may determine a look-ahead distance for the vehicle 200, which may have a lower limit range from 10 meters to 20 meters, and may be calculated as the product of the speed of the vehicle 200 and the determined look-ahead time. For example, as the speed of the vehicle 200 decreases, the look-ahead distance may also decrease (e.g., until the look-ahead distance reaches the lower limit). Additionally or alternatively, the processing unit 110 may execute the stereo image analysis module 404 to perform stereo image analysis of the first and second plurality of images to create a 3D map of the road ahead of the vehicle and detect features within the image, such as the object 1202. The processing unit 110 may estimate the position of the vehicle 200 relative to the object 1202 by observing differences that may exist relative to the object appearing in at least two of the plurality of captured images within the image data. For example, the position, speed, and / or acceleration relative to the vehicle 200 may be determined based on the trajectory, position, movement characteristics, etc. of features associated with the object 1202.

[0230] Object 1202 may be a relatively flat object, or it may be three-dimensional. As discussed in further detail below, if the object is flat or relatively flat (e.g., 2 cm or less, 1 cm or less, etc.), system 100 may determine that no further action or change in the route of vehicle 200 is required. Fig.14A In the example of , the object 1202 is again a cardboard box. In the first image 1402, the object 1202 has an initial dimension o x1 , o y1 and z1 System 100, via object detection module 1310, may be configured to determine these dimensions based on reference plane 1408, lane constraints of the detected road, or other reference points of known size and / or position within the image data. In a second image 1404 taken at a later time, object 1202 is then closer to the vehicle and appears larger. Object 1202 then has a second dimension o x2 , oy2 and z2 The system 100 may determine the second set of dimensions in the same manner as the first set of dimensions.

[0231] In some embodiments, the system 100 may use the determined dimensions of the object 1202 in the first image 1402 and at least the second image (such as image 1404) to gather more information about the object and determine one or more attributes of the object. In some embodiments, the system 100 (via the processing unit 110 and / or the image processor 190) may be configured to automatically recognize the object based on known objects previously programmed into an integrated database or a database accessible via the Internet. In these embodiments, the system 100 may be configured to generate audio or visual feedback to a human operator of the vehicle 200.

[0232] The system 100 may determine, for example, one or more dimensions from the image 1404. x2 , o y2 and z2 With dimension o from image 1402 x1 , o y1 and z1 If the second set of dimensions from image 1404 is less than the dimensions from first image 1402, then the object is likely moving away from vehicle 200 and is generally not of interest. If the second set of dimensions is determined to be greater than the first set of dimensions, then the determination may indicate that object 1202 is either stationary on the road or moving toward vehicle 200, and other actions may be taken. For example, system 100 may determine the height of object 1202 relative to reference plane 1408 (essentially, dimensions o x1 / o x2 ) to determine the approximate height of the object 1202 relative to the road surface. In some embodiments, the system 100 can be configured to detect objects as small as 2 cm high. Based on the determined height of the object, the system 100 can perform one of a plurality of alternative actions, as will be discussed in more detail below.

[0233] Fig. 14B Provided Fig.12 8. Vehicle 200 (here represented as vehicle 200c) is again traveling in lane 810 of road 800. Vehicle 200c is again equipped with image capture devices 122 and 124; more or fewer devices may be associated with any particular vehicle 200.

[0234] Object 1202 is located on lane 810 of road 800. Fig.14A (and combined with Fig.151500), the system 100 may be configured to determine a plurality of attributes of the object 1202 based on a plurality of images acquired from one or more image capture devices 122-126 and received via the data interface 128. These attributes may include the location and size of the object 1202, including its height relative to the surface of the road 800.

[0235] As an initial step, the system 100 may determine the relative position of the vehicle 200c and the object 1202 via the position sensor 130 and / or the lane positioning module 1320. This determination may include determining lane constraints or other such position indicators associated with the road 800 and / or the lanes 810 / 820. The system 100 may then determine the position of the vehicle 200c within its current lane of travel (here, lane 810) via the processing unit 110 by calculating the distance from the determined constraints. Fig. 14B In the example of , system 100 may, for example, determine distances q1 and q2 from one or more surfaces of vehicle 200 to lane constraints. Further, system 100 may determine distance d2 from the surface of vehicle 200c to the surface of object 1202. System 100 may also determine the relative positioning of object 1202 located laterally inside the lane, such as by measuring distance q3 from the surface of object 1202 to lane constraint separation lanes 810 and 820. Each of these determined distances (such as q1, q2, etc.) may be measured from any portion of the interior or exterior of the vehicle 200c, including, but not limited to, the front of the vehicle 200c, portions of the vehicle 200c such as headlights or front license plates, mounted locations of the image capture devices 122 or 124, a determined centroid of the vehicle 200c, the rear of the vehicle 200c, one or more windshields or mirrors associated with the vehicle 200c, wheels of the vehicle 200c, the right or left side or windows of the vehicle 200c, points associated with the roof of the vehicle 200c, or points associated with the chassis of the vehicle 200c. The distances d2 and q3 may also be measured as any portion of the object 1202 in three-dimensional space.

[0236] At least based on the above combination Fig.14A the size of the determined object 1202 in question, and Fig. 14B, the system 100 can determine one or more actions to be performed. The system 100 can perform actions via the action module 1330 to send information and instructions to various associated subsystems of the vehicle 200c to perform actions, which subsystems include, but are not limited to, the throttle system 220, the braking system 230, and the steering system 240. The actions performed by the action module 1330 can also be informed by the determination of the road clearance height of the vehicle 200c. The clearance height can be a known value stored in the memory 140 / 150 of the system 100, or it can be determined again in each driving situation based on various factors such as the inflation state of the tires, the road surface, weather conditions, etc. In some embodiments, the clearance height can be measured using optics and reference points (e.g., from the vehicle 200c to a point on the road surface) or by sound echo, light echo, radio frequency from the road surface, etc. The action module 1330 can then determine one of a plurality of alternative actions to be performed based on the determined height of the object 1202 and the road clearance height of the vehicle 200c. For example, if the height of object 1202 is less than the road clearance height of vehicle 200c, action module 1330 may determine that the safe course of action is to simply do nothing and allow vehicle 200c to drive past object 1202, which is positioned between the wheels of vehicle 200c. Alternatively, action module 1330 may determine that the height of object 1202 is less than the road clearance height of vehicle 200c, but a slight change in the directional course of vehicle 200c may allow vehicle 200c to safely drive past object 1202. Further, system 100 and action module 1330 may determine that the height of object 1202 exceeds the road clearance height of vehicle 200c, and that correct adjustments to the directional course of vehicle 200c may be required in order to avoid object 1202. These adjustments may also require instructions to throttle system 220 and / or brake system 230 in order to safely make the course change.

[0237] The latter scenario Fig. 14B This illustration is merely an example and is not intended to be limiting. Fig. 14B In the example of FIG. 1 , the system 100 has determined that the height of the object 1202 is higher than the road clearance height of the vehicle 200c. Therefore, the action module 1330 can determine that the best action path for the vehicle 200c is to move slightly to the right to avoid the object 1202 based on the detected lane constraints and the measured distance as described above. The action module 1330 can then send instructions to the throttle system 220, the braking system 230, and the steering system 240 to implement the change in the directional path.

[0238] When vehicle 200c passes object 1202, the projected position of vehicle 200c is represented by vehicle 200d. To successfully complete the passing of object 1202, system 100 may determine a new additional distance. For example, system 100 may determine Fig. 14B , which corresponds to the required safety distance from the object 1202 to the surface of the vehicle 200d. The system 100 can also calculate distances q5 and q6 via the lane positioning module 1320, which represent the position of the vehicle 200d relative to the detected lane constraints. These distances can be sent to the action module 1330 to help determine the appropriate directional route of the vehicle 200d. Ideally, the vehicle 200d will successfully turn through the object 1202 while staying within the constraints of the lane 810. However, in some embodiments, in order to avoid the object 1202, the vehicle 200d may need to move to the outside of the constraints of the lane 810, or the vehicle 200d may need to simply change lanes, such as changing to lane 820. Once the object 1202 has been passed, the action module 1330 can send instructions to the throttle system 220, the braking system 230, and the steering system 240 that can restore the normal operation of the vehicle 200.

[0239] Fig.15 A process 1500 for detecting an object on a road consistent with the disclosed embodiments is illustrated. The steps of the process 1500 may be performed by one or more of the processing unit 110, the image acquisition unit 120, the position sensor 130, the image processor 190, the object detection module 1310, the lane positioning module 1320, or the action module 1330.

[0240] At step 1510, the process 1500 may include: using at least one image capture device 122, 124 and / or 126 to obtain multiple images of the area near the vehicle 200. In some embodiments, the multiple images may be images of the area in front of the vehicle 200. The processing unit 110 may receive the multiple images from the (multiple) image capture devices via the data interface 128. The multiple images may then be processed in real time by the image processor 190 of the processing unit 110. The image processor 190 may process a first image captured by the at least one image capture device, and at least one additional image captured at a later point in time. The at least one image capture device may be configured to have a field of view between approximately 35 degrees and approximately 55 degrees. In combination with the above Fig.14A Examples of acquired images and potential fields of view for system 100 are illustrated.

[0241] At step 1520, process 1500 may determine a reference plane corresponding to the surface of road 800. This determination may be performed by one or more of image processor 190, object detection module 1310, or lane location module 1320. The reference plane may be combined with the above Fig.14A The reference plane may be similar to the plane 1408 described above. The reference plane may be determined based on a plurality of images captured by at least one image capture device. In these embodiments, the reference plane may be determined by analyzing a first image and at least a second image of the plurality of images. In some embodiments, the position of the road surface may be determined by analyzing the plurality of images by the image processor 190.

[0242] The reference plane may be detected by means of disparity information or by detecting one or more lane constraints on the road surface. In some embodiments, the reference plane may be determined by color difference or darkness difference in the image data. The processing unit 110 may generate and propose multiple potential reference planes using the image processor 190, and then select the best candidate. Once the reference plane is selected, information about its position may be stored inside the system 100, such as in the memory 140 / 150. In some embodiments, the processing unit 110 and / or the image processor 190 may periodically re-determine the reference plane based on changes in the road surface (such as changes in elevation or changes in curvature) or when the vehicle 200 turns to a different road. In some embodiments, the processing unit 110 may be configured to determine a reference plane 1408 corresponding to the surface of the road 800 as an average surface level associated with the road surface appearing in multiple images. This method may be useful, for example, when the road 800 is not completely flat or abnormal (such as if the road surface includes one or more of gravel, potholes, snow or ice, mud, vegetation, etc.).

[0243] At step 1530, process 1500 may locate the target object in a first image and at least one second image captured later within the plurality of images. Fig.14A As illustrated, the object detection module 1310 can locate the image based on its presence in multiple images among multiple images acquired by the system. The object detection module 1310 can determine one or more potential measurements associated with the detected object. FIG. 14A to FIG. 14BAs discussed, these measurements may include relative measurements of other objects, such as vehicles or lane constraints. Determination of the position of an object may be aided by direct measurement tools, physical devices, or modules, such as position sensors 130. In other embodiments, these determinations may include one or more visual determinations. Visual determinations may be determined directly by software instructions executed by the image acquisition unit 120 in a "rangefinder" manner, or may be determined from analysis of image data received by the image processor 190 from the data interface 128 and acquired by one or more image capture devices 122-126. Once determined, the position of the object may be stored within the system 100, such as in the memory 140 / 150, and may be periodically updated by the object detection module 1310.

[0244] At step 1540, the process 1500 may determine one or more attributes of the detected object(s). In some embodiments, the object may be identified as a specific object based on information stored in a database associated with the system 100. As discussed above, these determined attributes may include the size of the detected object (such as its height relative to the detected reference plane). The system 100 may be configured to determine the height of any object extending at least about 2 cm above the reference plane within the road. In an alternative embodiment, the system 100 may be configured to detect and determine the attributes of any object extending between 2 cm and 10 cm above the reference plane. In other embodiments, the determination of the height of the target object may also be based on the speed of the detected vehicle. The speed may be determined based on information received from one or more of the throttle system 220 or the brake system 230. Other details about calculating the reference plane for the road surface and calculating the height of the target object may be found in U.S. patent application 14 / 554,500 filed on November 26, 2014, the entire contents of which are incorporated herein by reference.

[0245] The system 100 can be configured to determine an action associated with the detected object based on the determined attributes. For example, the system 100 and the action module 1330 can be configured not to perform any substantial action unless the determined height of the detected object exceeds a threshold level. In some embodiments, as discussed above, the threshold level can be approximately 2 cm, or can be higher (e.g., 3 cm, 4 cm, 5 cm, 6 cm, etc.) depending on user or system preferences. For example, in some embodiments, the threshold can be associated with the detected road clearance height of the vehicle 200. If the processing unit 110 determines from the acquired information that the height of the object does not exceed the configured threshold (step 1550: No), the process 1500 can return to the beginning, and the system 100 can continue to monitor the status of potential objects within the road.

[0246] Alternatively, if the system 100 determines that action may be required in response to the presence of the detected object in the road (step 1550: yes), the process 1100 can proceed to step 1560. In step 1560, the process 1500 can cause the vehicle 200 to change the directional route to avoid or pass the detected object. The processing unit 110 can send instructions to one or more systems associated with the vehicle 200 via the action module 1330. For example, an instruction can be sent to the steering system 240, instructing the vehicle 200 to merge into the next lane to the left or right. Alternatively, an instruction can be sent to turn around the object, or change the directional route so that the vehicle only drives over the object, which is located between the wheels. Other instructions can be sent; for example, the image processor 190 can determine from the real-time image data that traffic or other obstacles prevent the immediate change of the directional route. In these embodiments, the instruction can first be sent to the braking system 230 and / or the turn signal, and the vehicle 200 can be slowed down or stopped until the directional route change can be made. At this point, it is safe to make a command to the steering system 220 and the vehicle can turn to change course.

[0247] In some alternative embodiments, where the vehicle 200 is partially or fully under the control of a human operator, step 1560 may instead include generating feedback to the operator notifying that an object has been detected on the road and that a change in directional route is suggested or required. In some embodiments, if the human operator does not respond to the feedback within a certain time period, an automated system such as system 100 and its action module 1330 may take over operation of the vehicle and cause the vehicle to automatically change route as described above. The time period may be predetermined, or may be based on a predetermined time period, such as in conjunction with the above. Fig.12 and FIG. 14A to FIG. 14B The described visual determination is calculated by visual determination.

[0248] Extra-wide traffic light detection system

[0249] In some embodiments, the system 100 can identify objects that are outside the line of sight of a typical driver position in the vehicle 200. The typical driver position in the vehicle 200 is located to the left of the front seat in countries driving on the right hand side of the road, and to the right of the front seat in countries driving on the left hand side of the road. The National Highway Traffic Safety Administration recommends that the driver himself be 10 feet (25.4 cm) away from the driver's airbag for safety reasons. This distance is measured from the center of the steering wheel to the driver's sternum. Because the vehicle 200 is opaque, parts of the vehicle 200 (e.g., the roof, doors, windshield frame, etc.) can blur the perspective starting from the typical driver position. In addition, although humans have a horizontal field of view facing forward of almost 180 degrees, the vertical range of the human field of view is significantly smaller. However, the driver assistance system can use an image capture device (e.g., image capture device 122) with an extended perspective that exceeds the perspective of the driver sitting on the typical driver position. The extended perspective can enable the system 100 to identify objects in the environment of the vehicle 200 that the driver cannot detect. Moreover, the system 100 can determine changes associated with the identified objects and act accordingly. For example, the system 100 may determine a change in state of a traffic light in an area above the vehicle 200. As another example, the system 100 may determine the trajectories of different objects (eg, pedestrians) in an area beside the vehicle 200.

[0250] Fig.16 and Fig.17 2 is a schematic representation of a vehicle 200 in two situations consistent with the disclosed embodiments. Fig.16 and Fig.17 As illustrated in FIG. 1 , the vehicle 200 may include an image capture device 122 associated with a first viewing angle. The first viewing angle is represented by FOV 1610 in the vertical direction, as shown in FIG. Fig.16 , and horizontally represented by FOV 1710, as shown in Fig.17 as shown in . Fig.16 and Fig.17 Also illustrated in FIG. 1 is a second perspective from a typical driver position in vehicle 200. The second perspective is represented by FOV 1620 in the vertical direction, as shown in FIG. Fig.16 , and represented by FOV1720 in the horizontal direction, as shown in Fig.17. Consistent with embodiments of the present disclosure, a first viewing angle associated with the image capture device 122 may be greater than a second viewing angle associated with a typical driver position in the vehicle 200. The increment between the larger first viewing angle and the second viewing angle is labeled as an extended area in the present disclosure. In some embodiments, the extended area may include objects that the system 100 may identify as being outside the line of sight of a typical driver position in the vehicle 200. Additionally, the system 100 may trigger one or more actions (e.g., generate a navigation response) based on determined changes associated with objects identified in the extended area.

[0251] Fig.16 A first example for determining changes associated with one or more objects located outside of the line of sight of a typical driver position of the vehicle 200 is illustrated. Fig.16 The angular ranges of FOV 1610 and FOV 1620 are used in FIG. 1 for illustrative purposes. In some embodiments, FOV 1620 may be, for example, approximately 65 degrees and FOV 1610 may be, for example, greater than 65 degrees. In other embodiments, FOV 1620 may be, for example, approximately 135 degrees and FOV 1610 may be, for example, greater than 150 degrees, greater than 160 degrees, or greater than 170 degrees. In some embodiments, FOV 1610 and FOV 1620 may be measured in an upward direction from a horizontal plane. Fig.16 As shown in , the extended area may include a traffic light pole 1630 located in an area above the vehicle 200. The traffic light pole 1630 located outside the line of sight of a typical driver position displays a red light. In some embodiments, the system 100 is capable of detecting a traffic light and a change in the display color of the traffic light without the driver of the vehicle 200 even being able to see the traffic light pole.

[0252] One example of these situations is when the driver of vehicle 200 applies the brakes in a manner that causes vehicle 200 to stop at the stop line of an intersection and / or places vehicle 200 too close to traffic light pole 1360. Another example of these situations is when vehicle 200 reaches the stop line at an intersection and traffic light pole 1630 is placed on a high pole so it can be seen from a distance. Because traffic light pole 1630 is located outside of FOV 1620, a driver sitting in a typical driver position will not be able to detect the state change when the light in traffic light pole 1630 turns green. However, because traffic light pole 1630 is located inside FOV and therefore taking into account image capture device 122, system 100 will be able to detect the state change of the traffic light. Identifying traffic lights in an area above vehicle 200 can improve safety and driver convenience. For example, when considering the safety of the driver and passengers of the vehicle 200, if the traffic light pole 1630 is not within the FOV 1620, it may be unsafe to back up the vehicle 200 until the traffic light pole 1630 is within the FOV 1620. In addition, even if it is not dangerous to back up the vehicle 200, it may be inconvenient for the driver to back up the vehicle 200. In one embodiment, after determining that the light in the traffic light pole 1630 has turned green, the system 100 may cause the vehicle 200 to accelerate to pass through the intersection.

[0253] Fig.17 A second example for determining changes associated with objects located outside of the line of sight of a typical driver position of vehicle 200 is illustrated. Fig.17 The angular ranges of FOV 1710 and FOV 1720 are used for illustration purposes. In some embodiments, FOV 1720 may be, for example, approximately 180 degrees, and FOV 1710 may be, for example, greater than 200 degrees, greater than 240 degrees, or greater than 300 degrees. Fig.17As shown in , the extended area may include pedestrians 1730 in the area next to the vehicle 200. The pedestrians 1730 may move in a specific direction. In some cases, the pedestrians 1730 may be positioned at the edge of the FOV 1720, but because the driver is focused on the road ahead, it may not be able to detect the pedestrians 1730. In contrast, the system 100 not only has a larger FOV, it does not have to focus on the road ahead and can identify objects located in any area of ​​its FOV. Because the pedestrian 213 is located within the FOV 1710, the system 100 can detect the pedestrians 1730 and determine the walking direction trajectory of the pedestrians. In other embodiments, the system 100 can identify objects located next to the vehicle 200, determine the moving direction trajectory of the objects, determine whether the objects are on a possible collision route with the vehicle 200, and thus act. For example, after determining that the pedestrians 1730 are on a possible collision route with the vehicle 200, the system 100 can control the horn of the vehicle to generate a type of audio alarm.

[0254] As discussed above, FIG. 3B to FIG. 3D An exemplary camera mount 370 is illustrated that is configured to be positioned behind the rearview mirror 310 and against the vehicle windshield. FIG. 3B to FIG. 3D As shown in , the camera mount 370 may include multiple image capture devices (122, 124, and 126). In one embodiment, the combined viewing angles of the image capture devices 122, 124, and 126 form a second viewing angle. For example, in order to make the FOV 1710 higher than 300 degrees, a combination of six image capture devices may be used. In various embodiments, FIG. 3B to FIG. 3D At least one of the image capture devices described in (e.g., image capture device 122) includes an ultra-wide lens. The ultra-wide lens may enable image capture device 122 to have a first viewing angle. In some embodiments, the ultra-wide lens may have a focal length that is shorter than the short side of the image sensor of image capture device 122. Alternatively, the ultra-wide lens may have a focal length that is shorter than the long side of the image sensor but longer than the short side of the image sensor. Consistent with embodiments of the present disclosure, image capture device 122 may be configured to capture an image of a traffic light in a conic section region that is positioned, for example, between 150 and 180 degrees, between 150 and 190 degrees, between 160 and 180 degrees, or between 170 and 180 degrees relative to the positioning of image capture device 122.

[0255] Fig.18 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations consistent with the disclosed embodiments. Fig.18As shown in , the memory 140 or 150 may store an object identification module 1802, a state determination module 1804, and a system response module 1806. The disclosed embodiments are not limited to any particular configuration of the memory 140 or 150. Further, the processing unit 110 may execute instructions stored in any module 1802-1806 (included in the memory 140 or 150).

[0256] In one embodiment, object recognition module 1802 may store software instructions (such as computer vision software) that, when executed by processing unit 110, identify objects that are outside the line of sight of a typical driver position in vehicle 200. In one embodiment, processing unit 110 may scan multiple images, compare portions of the images to one or more predetermined patterns, and identify traffic lights in the area next to vehicle 200. In another embodiment, processing unit 110 may scan multiple images, compare portions of the images to one or more predetermined patterns, and identify objects that are located in the area next to vehicle 200. In some embodiments, module 1802 may store software instructions that, when executed by processing unit 110, identify intersections and determine that traffic lights may be nearby. One way that processing unit 110 identifies intersections is by using historical image data. For example, processing unit 110 may determine that a traffic light may be nearby even when a traffic light is not seen in the current image, but was seen in a previous image (e.g., a previous frame, five frames ago, etc.). Additionally or alternatively, processing unit 110 may identify image data objects that indicate a nearby intersection. For example, processing unit 110 may determine that a traffic light may be nearby by identifying a crossroad in image data of the road ahead. In another example, processing unit 110 may identify in the image data the rear of other traffic lights that control traffic going in a direction opposite to the direction of travel of vehicle 200. Additionally or alternatively, processing unit 110 may use map data (such as data from map database 160) to identify the intersection. For example, processing unit 110 may consider a typical road shape description extracted from map database 160 to determine that a traffic light may be nearby.

[0257] In another embodiment, module 1802 may store software instructions that, when executed by processing unit 110, correct optical distortion in multiple images to enable identification of objects. The optical distortion may be the result of the image capture device 122 using an ultra-wide lens. For example, the optical distortion may be a barrel distortion in which the image magnification decreases with the distance from the optical axis, and thus the distortion may be closest to the edge of the image frame. Consistent with the present disclosure, multiple algorithms may be used to correct optical distortion that may be produced using an ultra-wide lens. For example, an algorithm may be used to correct optical distortion to identify an object close to the edge of an image frame. As another example, different algorithms may be used to identify the state of an object located at the edge of an image frame. In another embodiment, the image capture device may include multiple image capture devices (e.g., image capture devices 122, 124, and 126), and each image capture device may be associated with a different field of view. Module 1802 may store software instructions that, when executed by processing unit 110, may correctly combine images derived from multiple image capture devices. For example, different portions of an object may be captured by different image capture devices even when there is some overlap in the fields of view of the two image capture devices. Thus, the object recognition module 1802 may include instructions for recognizing objects in areas spanning adjacent images captured by different image capture devices.

[0258] In one embodiment, the state determination module 1804 may store software instructions that, when executed by the processing unit 110, determine the state of an object identified in the image data and located at least partially within the extended area. Consistent with the present disclosure, the extended area includes an area between the field of view of the image capture device (e.g., FOV 1610 and 1710) and the field of view of a typical driver position of the vehicle 200 (e.g., FOV 1620 and 1720). In one embodiment, the state determination module 1804 may store software instructions that, when executed by the processing unit 110, determine the state of a traffic light located in an area above the vehicle 200. For example, the processing unit 110 may determine the state of the traffic light by comparing one or more images to a stored pattern. In addition, a change in the state of the traffic light may be determined based on a difference between the stored pattern and two or more images taken at different times. In another embodiment, the state determination module 1804 may store software instructions that, when executed by the processing unit 110, determine the trajectory of a moving object located in an area next to the vehicle 200. For example, the processing unit 110 may determine the trajectory of the moving object by comparing the plurality of images with each other. In addition, the processing unit 110 may determine whether the moving object is located on a possible collision path with the vehicle 200.

[0259] In one embodiment, the system response module 1806 may store software instructions that, when executed by the processing unit 110, generate one or more responses in the vehicle 200. One type of system response may include a navigation response, as described in detail above with reference to the navigation response module 408. Other types of system responses may involve controlling the throttle system 220, the braking system 230, and / or the steering system 240. For example, the processing unit 110 may send an electronic signal causing the system 100 to physically depress the brakes by a predetermined amount or partially release the accelerator of the vehicle 200. Further, the processing unit 110 may send an electronic signal causing the system 100 vehicle 200 to turn in a particular direction. Such a response may be based on a determination of an associated traffic light. Other system responses may include providing various notifications (e.g., warnings and / or alarms). Notifications may include: reporting the status of a traffic light located above the vehicle 200; reporting a change in the status of a traffic light located above the vehicle 200; reporting the presence of an object next to the vehicle; and reporting that an object is located on a collision path with the vehicle 200.

[0260] Fig.19 is shown for use based on Fig.16 Example process 1900 for navigating a vehicle based on changes in the state of a traffic light in an area above the vehicle 200 as described in FIG. Fig.17 Thus, the exemplary process 1900 can be applied to various situations, including Fig.17 The situation described in .

[0261] like Fig.19 As shown in , in step 1902, the image capture device 122 may acquire multiple images of traffic lights located in an area above the vehicle 200 outside the line of sight of a typical driver position. In step 1904, the processing unit 110 may determine the association of the captured traffic lights in the multiple images based on at least one indicator of the vehicle position (e.g., lane markings). In some embodiments, this step may be optional and may be omitted. In step 1906, the processing unit 110 may determine the state of the traffic light based on an analysis of at least a first image from among the multiple images. In step 1908, the processing unit 110 may determine a change in the state of the traffic light based on at least a second image from among the multiple images. For example, the change in the state of the traffic light may be from a red light to a green light. After determining that the state of the traffic light has changed, in step 1910, the processing unit 110 may generate a system response based on the determined state change. These steps of process 1900 will be discussed in more detail below.

[0262] At step 1902, image capture device 122 may acquire a plurality of images of a traffic light located in an area above vehicle 200. Consistent with disclosed embodiments, image capture device 122 may include one or more image capture devices. Consistent with some embodiments, the traffic light may be located outside of the line of sight of a typical driver position in vehicle 200. For example, the traffic light may be located as shown in FIG. Fig.16 In some embodiments, the image capture device 122 may be located inside the vehicle 200. In addition, although the image capture device 122 may be located behind the rearview mirror and against the vehicle windshield in some embodiments, the processing unit 110 may determine the state of the traffic light through the windshield.

[0263] At step 1904, the processing unit 110 may determine the association of traffic lights in the area above the vehicle 200 based on at least one indicator of the vehicle location. A detailed disclosure of how the system 100 may determine the association of traffic lights will be described below in conjunction with Figure 20 to Figure 23 to be given. In some embodiments, an indicator of the vehicle position can be derived from an analysis of multiple images. For example, the processing unit 110 can determine the relative position of the traffic light to the vehicle 200 based on the identification of lane markings in the multiple images. Additionally or alternatively, the indicator of the vehicle position can be derived from geographic location data retrieved from the position sensor 130. For example, the processing unit 110 can determine the relative position of the traffic light to the vehicle 200 based on a GPS signal or a local positioning signal. In some embodiments, the processing unit 110 can compare the vehicle position obtained by GPS with the map data to determine the association of the traffic light. For example, the processing unit 110 can use information derived from the image data and the map data for analysis. Using the derived information, the processing unit 110 can determine the correspondence between the detected traffic light in the area above the vehicle 200 and the lane in which the vehicle 200 is currently traveling.

[0264] In step 1906, the processing unit 110 may determine the state of the traffic light based on an analysis of at least the first image from among the plurality of images. In some embodiments, the state of the traffic light in the area above the vehicle 200 is associated with the color indicated by the traffic light. For example, a steady green light may indicate that the vehicle 200 can continue after giving way to other vehicles, bicycles, or pedestrians on the road; a flashing yellow light is a warning and may indicate that the vehicle 200 should proceed with caution; and a steady red light may indicate that the vehicle 200 needs to stop. Additionally or alternatively, the state of the traffic light may also include additional information associated with the traffic light. The additional information may be the type of traffic light. For example, the traffic light in the area above the vehicle 200 may be a pedestrian crossing signal. Therefore, the processing unit 110 may determine that a pedestrian may pass the road in front of the vehicle 200. In some embodiments, in order to determine the state of the traffic light in the area above the vehicle 200, the processing device 110 may correct optical distortion in at least the first image. When the image capture device 122 includes an ultra-wide lens, optical distortion may occur.

[0265] In step 1908, the processing unit 110 may determine a change in the state of the traffic light based on at least a second image from among the multiple images. In some embodiments, the state of the traffic light in the area above the vehicle 200 may, for example, change from a red light to a green light. In one embodiment, at least the second image may include two or more images following at least the first image. In addition, the processing unit 110 may process only a portion of at least the second image. For example, when the vehicle 200 is not moving, the position of the traffic light in the frame remains the same. Therefore, in some embodiments, after identifying a portion of the frame including the traffic light pole in at least the first image, the processing unit 110 may determine the change in state by processing only the same portion of the frame in at least the second image. For example, in some embodiments, by processing only a portion of the frame including the traffic light pole, the processing unit 110 may save processing power.

[0266] In step 1910, the processing unit 110 may generate a system response based on the determined change in state. In some embodiments, the processing unit 110 may generate one or more system responses (e.g., two or more responses), including different types of responses. One type of system response may include providing various notifications (e.g., warnings and / or alarms) to the driver of the vehicle 200. The notification may be provided via the speaker 360 or the touch screen 320. For example, the system response may include reporting the change in state to the driver of the vehicle 200. Another type of system response may include a navigation response, which may include, for example, accelerating to a cruising speed specified by the driver.

[0267] Identify relevant traffic lights

[0268] In some embodiments, the system 100 can distinguish between associated and non-associated (or less associated) traffic lights. During a typical driving process, the vehicle 200 may pass through one or more hubs (such as an intersection with multiple traffic lights). For example, one or more traffic lights at an intersection may control traffic traveling toward the intersection in a particular direction. Therefore, one or more traffic lights may control whether the vehicle 200 can continue to travel through the intersection or whether the vehicle 200 must stop at the intersection. However, in addition to the traffic lights controlling the lane in which the vehicle 200 is traveling, other traffic lights controlling traffic on other lanes are also visible to the vehicle 200. Navigating the vehicle 200 based on any of the traffic lights controlling the lanes other than the lane in which the vehicle 200 is traveling may produce a navigation response that is undesirable or inappropriate for the planned route of the vehicle 200. Therefore, in order to enable self-control of the vehicle 200 in a manner suitable for the planned navigation path of the vehicle 200, the system 100 can identify which of the multiple traffic lights is controlling the traffic in the lane where the vehicle 200 is traveling while ignoring (or paying less attention to) other traffic lights that control other traffic lanes. Further, after the system 100 identifies the associated traffic light, the system 100 can identify the state of the traffic light (e.g., red, yellow, green) and implement an appropriate navigation response. For example, when a red light that controls the lane where the vehicle 200 is traveling is identified, or when a yellow light that controls the lane where the vehicle 200 is traveling is identified and the vehicle 200 passes a predetermined distance from the hub, the system 100 can suspend cruise control and apply braking.

[0269] Distinguishing between relevant and non-relevant (or less relevant) traffic lights on a road is complex. Fig. 20 and Fig.21 Examples of situations in which driving based on information derived from inapplicable traffic lights may be undesirable are illustrated. Fig. 20 In FIG. 1 , vehicle 200 a is traveling on a multi-lane road. Each lane of the road is associated with a different traffic light frame. Vehicle 200 a is approaching an intersection and is traveling in a lane designated for proceeding through the intersection and reaching the other side of the intersection. Similarly, Fig. 20 As shown in FIG, vehicle 200b is traveling in a lane that allows traffic to continue straight and pass through the intersection or allows traffic to make a right turn. The traffic light frame associated with the lane in which vehicle 200b is traveling includes a right turn traffic light. As another example, in FIG. Fig.21In FIG. 2 , vehicle 200a arrives at a hub of a non-perpendicular road. Multiple traffic light poles (eg, traffic light poles 2122 and 2124 ) including some traffic light poles that do not control the lane in which vehicle 200a is traveling are visible to vehicle 200a due to the positioning of the hub.

[0270] return Fig. 20 , an intersection 2000 with the following driving options is shown: road A with lanes leading to roads B, D, and F; road C with lanes leading to roads D and F; road E with lanes leading to roads F, H, and B; and road G with lanes leading to roads H, B, and D. Road A is associated with four traffic light poles 2012, 2014, 2016, and 2018. In the illustrated case, each traffic light pole controls a different lane. Road C is associated with two traffic light poles 2022 and 2024. Traffic light pole 2024 includes a traffic light 2024a for continuing straight and a traffic light 2024b for turning right (e.g., a green arrow is displayed when a right turn is authorized). Road E is associated with three traffic light poles 2032, 2034, and 2036. And road G is associated with two traffic light poles 2042 and 2044. The traffic light frame 2042 includes a traffic light 2042a for continuing straight driving and a traffic light 2042b for turning left. Fig. 20 In the illustrated case, traffic light 2042b and traffic light poles 2012, 2018, and 2032 display green lights, while all other traffic light poles display red lights. Fig. 20 There may be many other road variations and associated traffic light configurations besides those shown in FIG.

[0271] exist Fig. 20 In the illustrated case, vehicle 200a is located on road A in a lane to continue straight through intersection 2000 to road D. However, the field of view of the image capture device included in vehicle 200a may include both traffic light pole 2012 and traffic light pole 2014 (or even other poles). Vehicle 200a arrives at intersection 2000 when both traffic light poles 2012 and 2014 are displaying red lights, and only the light in the nearest traffic light pole 2012 has turned green. In this case, it is important that the traffic light detection system 100 of vehicle 200a recognizes that the green light of pole 2012 is not applicable to vehicle 200a. However, system 100 should base the determined navigation response on the state of the light associated with the other associated pole 2014.

[0272] exist Fig. 20On the other hand, in the case illustrated in FIG. 2 , vehicle 200b is located in a lane on road C that continues straight through intersection 2000 to road F, and that also allows traffic to make a right turn to road D. Vehicle 200b faces a traffic light pole 2024 that includes a traffic light 2024a for continuing straight and a traffic light 2024b for turning right. Vehicle 200b arrives at intersection 2000 when traffic light pole 2024 displays red lights for continuing straight (traffic light 2024a) and for turning right (traffic light 2024b), and only recently traffic light 2024b has turned green. This means that the current state of traffic light pole 2024 prevents traffic from traveling straight and turning right. If vehicle 200b acts on the state of light 2024b without recognizing and taking into account the state of light 2024a, an undesirable situation may occur. For example, if the vehicle 200b travels to the road F (ie, travels straight) based on the information of the traffic light 2024b (showing a green light), a vehicle traveling from the road E to the road B may create a danger to the vehicle 200b.

[0273] Steering Fig.21 , Fig.21 An intersection 2100 is illustrated with non-perpendicular roads with the following travel options: Road A with two lanes, one lane leading to Road C and the other lane leading to Road E; Road B with two lanes leading to Road C and one lane also leading to Road E; and Road D with a single lane leading to Road E. Road A is associated with two traffic light poles 2112 and 211, and each traffic light pole controls a different lane. Road B is also associated with two traffic light poles 2122 and 2124, and each traffic light pole controls a different lane. Road D is associated with a single traffic light pole 2132. Fig.21 In the illustrated case, traffic light pole 2122 displays a green light, while all other traffic light poles display a red light. Fig.21 , vehicle 200a is located on road A in a lane that continues straight through intersection 2000 to road C. However, the field of view of the image capture device 122 included in vehicle 200a may include both traffic light pole 2114 and traffic light pole 2122. In this example, vehicle 200a arrives at intersection 2100 when both traffic light poles 2114 and 2122 are red, and only the light in traffic light pole 2122 has recently turned green. In this case, it is important that vehicle 200a determines its navigation response based on the state of traffic light pole 2114, rather than based on the state of traffic light pole 2122.

[0274] Fig. 20 and Fig.21The three scenarios described in provide only some examples of road conditions in which it may be helpful to have a driver assistance system that can distinguish between associated and non-associated traffic light poles, to determine the state of the traffic lights included in the associated traffic light poles, and to generate and make an appropriate navigation response based on the state. Further, these examples demonstrate that the system may need to evaluate multiple traffic light poles facing the vehicle in order to identify the traffic light pole that is most appropriate for the lane in which the vehicle is traveling or the planned direction of travel of the vehicle (especially in situations where, for example, multiple traffic lights or traffic light poles can be associated with a single lane in which the vehicle is traveling).

[0275] As discussed above, the system 100 can distinguish between associated and non-associated traffic light poles, determine the status of traffic lights included in (multiple) associated traffic light poles, and generate and make appropriate navigation responses based on the status in various driving scenarios. For example, when the vehicle 200 approaches an intersection, the system 100 can determine which traffic light is associated, determine the status of the traffic light, and discover any other associated information in the image captured by one or more image capture devices 122-126. If the traffic light is red, the system 100 can cause the vehicle 200 to apply its brakes. If the traffic light is green, the system 100 can cause the vehicle 200 to continue. If the traffic light is yellow, the system 100 can determine the distance to the intersection and / or the estimated time to the intersection based on analysis of the image, the speed of the vehicle 200, and / or location data (e.g., GPS data). If the vehicle 200 is within a predetermined time (e.g., five seconds, ten seconds, etc.) and / or distance (e.g., one meter, five meters, ten meters, etc.) threshold, the system 100 may cause the vehicle 200 to continue. If the vehicle 200 is not within the predetermined time threshold and / or distance threshold, the system 100 may cause the vehicle 200 to stop before reaching the intersection. As another example, when the vehicle 200 is stopped at a traffic light, after the traffic light changes its state from red to green, the system 100 may generate a navigation response that includes, for example, applying the accelerator, releasing the brake, and turning through the intersection.

[0276] Fig. 22is an exemplary block diagram of a memory 140 / 150 that is configured to store instructions for performing one or more operations consistent with the disclosed embodiments. The memory 140 / 150 may be accessed by the processing unit 110. As discussed above, the processing unit 110 may include various devices, such as a controller, an image preprocessor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, a memory, or any other type of device for image processing and analysis. Thus, the memory 140 / 150 may be accessed by the processing unit 110, may be integrated with the processing unit 110, or may be included in an embedded system with the processing unit 110. Fig. 22 As shown in , the memory 140 or 150 may store the association determination module 2202, the state determination module 2204, and the system response module 2206. The disclosed embodiments are not limited to any particular configuration of the memory 140. Further, the processing unit 110 may execute instructions stored in any module 2202-2206 (included in the memory 140 or 150).

[0277] In one embodiment, the association determination module 2202 may store software instructions (such as computer vision software) that, when executed by the processing unit 110, determine the association of each of the plurality of traffic light poles with the vehicle 200. For purposes of the present disclosure, the step of determining the association of the traffic light poles may include performing one or more evaluations. In one embodiment, the processing unit 110 may evaluate the orientation of each of the plurality of traffic light poles relative to the vehicle. For example, in Fig.21 In the illustrated case, the field of view of the image capture device 122 includes both the traffic light pole 2114 and the traffic light pole 2122. However, the orientation of the traffic light pole 2114 indicates that the traffic light pole 2114 is more associated with the vehicle 200a than the traffic light pole 2122.

[0278] Other evaluations may be applicable to the decision making process. For example, the processing unit 110 may evaluate the distance of each of a plurality of traffic light poles relative to the vehicle. Additionally or alternatively, the processing unit 110 may evaluate which traffic light is facing the front of the vehicle 200. Additionally or alternatively, the processing unit 110 may use the identified lane markings to divide the area in front of the vehicle 200 into a plurality of regions, associate each identified traffic light pole with a region, and evaluate which region is most relevant to the vehicle 200. Additionally or alternatively, the processing unit 110 may compare the vehicle position obtained via GPS with map data to determine the association of traffic lights. For example, the vehicle 200 may evaluate map data including information about possible travel options at a plurality of locations. Using the vehicle position obtained by GPS, the processing unit 110 may determine which travel options are available to the vehicle 200 approaching the hub, and use this information to determine the association of the traffic light at the hub with the vehicle 200.

[0279] In one embodiment, the state determination module 2204 may store software instructions that, when executed by the processing unit 110, determine the state of a traffic light included in at least one traffic light frame determined to be associated with the vehicle. As described above, the state of a traffic light may also be associated with information provided by the traffic light. In a typical traffic light, the displayed color and relative position of the illuminated traffic light may provide basic information associated with the vehicle 200.

[0280] The state determination module 2204 can derive additional information from the environment of the associated traffic light pole and from the atypical traffic lights included in the associated traffic light pole. For example, the associated traffic light pole may have a sign near it that includes associated text, such as a sign stating a specific date and time. Therefore, in some embodiments, the state determination module 2204 can derive information from the text included in the sign associated with the associated traffic light pole. For example, the state determination module 2204 can implement optical character recognition technology to identify the text in the sign. The state determination module 2204 can then compare the identified text with a database to determine the information provided by the sign. As another example, the associated traffic light pole may include a pedestrian passing signal. The state determination module 2204 can determine that the state of the pedestrian passing signal means that pedestrians can cross the road in front of the vehicle 200 while the vehicle 200 has a green light to turn right.

[0281] In one embodiment, the system response module 2206 may store software instructions (such as computer vision software) that, when executed by the processing unit 110, generate one or more responses in the host vehicle 200. One type of system response may include one or more navigation responses, as described in detail above with reference to the navigation response module 408. Other types of system responses may provide control signals to the throttle system 220, the braking system 230, and / or the steering system 240. For example, the processing unit 110 may send an electronic signal causing the system 100 to physically apply the brakes to a predetermined amount or partially release the accelerator of the vehicle 200. Further, the processing unit 110 may send an electronic signal causing the system 100 to turn the vehicle 200 in a particular direction. Such a response may be based on a determination of an associated traffic light. Further, if a pedestrian may cross the road in front of the vehicle 200, the system response may include performing additional processing of the image data. For example, the processing unit 110 may confirm that pedestrians near the vehicle 200 are not walking on a trajectory that would put the pedestrians in danger while the vehicle 200 is moving.

[0282] The location of the traffic light pole in the hub can be before, after, or in the middle of the hub. Identifying the location of each traffic light pole in the hub can be used, for example, to determine the association of the traffic light poles. In some embodiments, the processing unit 110 can estimate the distance of one or more traffic light poles relative to the vehicle 200 to create a 3D model of the hub. In one embodiment, the 3D model of the hub can be stored for future use. The 3D model of the hub can include one or more of the following: the 3D location of one or more traffic light poles, the relative distance between each traffic light pole in the hub and other traffic light poles, the (multiple) directions indicated by each traffic light pole, and the relative distance between each traffic light pole and the stop line of the hub. In addition, when the vehicle 200 approaches the hub, the 3D model can be updated regularly using identifiable details. Examples of identifiable details can include arrows in traffic lights, lane markings in the hub, and the like. In one example, when the vehicle 200 passes through the hub, the identified details are compared with the information stored in the 3D model, and if appropriate, the 3D model is updated.

[0283] Consistent with embodiments of the present disclosure, the 3D model may be used to determine the association of one or more traffic light poles. In order to utilize sufficient time to activate the brake system 230 to stop the vehicle 200 before the hub, the system 100 may determine which traffic light is associated with the vehicle 200 at a distance of approximately eighty meters from the hub. As the vehicle 200 approaches the hub, information derived from the captured image may be compared with the 3D model to find a match. For example, the processing unit 110 may compare the relative distances between the identified traffic light poles in the hub with the 3D model to determine the association of one or more traffic light poles. Using the 3D model, the processing unit 110 may identify the associated traffic light when the vehicle 200 is more than 50 meters, 75 meters, 100 meters, or 125 meters away from the hub. As another example, the processing unit 110 may compare the relative distances between the identified traffic light poles in the hub with the 3D model to determine the distance to the stop line of the hub, even when the stop line is not visible from the current position of the vehicle 200. Using the 3D model, the processing unit 110 can determine the distance to the stop line of the hub when the vehicle 200 is more than 50 meters, 75 meters, 100 meters, or 125 meters away from the hub. Fig.23 Further details are provided regarding the relevance determination module 2202 , the state determination module 2204 , and the system response module 2206 .

[0284] Fig.23 2300 is a flow chart illustrating an exemplary process 2300 for identifying associated traffic lights consistent with disclosed embodiments. The system 100 may implement the process 2300 to process, for example, Fig. 20 and Fig.21 The situation described in .

[0285] like Fig.23 As illustrated in FIG. 23, at step 2310, the image capture device 122 may acquire an image of an area in front of the vehicle 200. The area may include a plurality of traffic light poles, each of which includes at least one traffic light. At step 2320, the processing unit 110 may determine at least one indicator of the vehicle's position, for example, based on features (e.g., lane markings) identified from the acquired image. At step 2330, the processing unit 110 may determine an association of each of the plurality of traffic light poles using the at least one indicator. At step 2340, the processing unit 110 may determine a state of a traffic light included in the associated traffic light pole. For example, the state of the traffic light may indicate that the vehicle 200 should stop (e.g., due to a red light) or can turn left (if there are no pedestrians passing by). After determining the state of the traffic light included in the traffic light pole, at step 2350, the processing unit 110 may generate a system response based on the determined state change. These steps of process 2300 will be discussed in more detail below.

[0286] For example, at step 2310, the image capture device 122 may acquire at least one image of an area in front of the vehicle 200. The “area in front of the vehicle” includes a geographic area located in front of the vehicle relative to its direction of movement. The area may include a hub (e.g., Fig. 20 In the middle is the intersection 2000 and in Fig.21 2100), crossroads, intersections, roundabouts, streets, roads, etc. In some cases, an area may include multiple traffic light frames, each of which includes at least one traffic light. A "traffic light frame" includes any form of structure that accommodates one or more light generating devices used to control traffic and / or provide road-related information. In some cases, two or more traffic light frames may be combined in a single traffic light assembly, but each traffic light frame may be associated with a different lane. A typical traffic light frame may include three circular traffic lights: a green traffic light, a yellow traffic light, and a red traffic light. A "traffic light" includes a device having at least one light source capable of displaying a unique color. In some cases, vehicle 200 may encounter an atypical traffic light frame. An atypical traffic light frame may include one or more non-circular traffic lights with different colors. For example, a right turn arrow traffic light, a left turn arrow traffic light, a public transportation traffic light, a pedestrian passing traffic light, a bicycle passing traffic light, etc.

[0287] In some embodiments, the processing unit 110 may receive at least one acquired image via the data interface 128. Figure 5D As discussed in detail, the processing unit 110 can identify a traffic light in at least one of the acquired images. For example, the processing unit 110 can filter the objects identified in at least one of the acquired images to construct a set of candidate objects, excluding those objects that are unlikely to correspond to traffic lights. The filtering can be performed based on various attributes associated with the traffic light, such as shape, size, texture, position (e.g., relative to the vehicle 200), etc. In addition, the processing unit 110 can analyze the geometry of the area in front of the vehicle 200. The analysis can be based on any combination of: (i) the number of lanes detected on either side of the vehicle 200; (ii) markings detected on the road (such as arrow markings); and (iii) a description of the area extracted from the map data. For example, if lane markings defining a lane of travel are identified on the road, and the traffic light pole is within the boundaries of the lane markings, the system 100 can conclude that the traffic light pole is associated with the lane associated with the lane markings.

[0288] At step 2320, the processing unit 110 may determine at least one indicator of the vehicle's location from the image. An "indicator of the vehicle's location" includes any form of information related to the physical location of the vehicle. The indicator of the vehicle's location may be derived from an analysis of image data (e.g., at least one acquired image). In addition, the indicator of the vehicle's location may be derived from geographic location data (e.g., GPS signals, local positioning signals, and / or map data) or from data indicating the location of the vehicle 200 relative to other vehicles on the road. In some embodiments, the indicator of the vehicle's location may include a distance from at least one traffic light pole derived from at least one image. In other embodiments, the at least one indicator of the vehicle's location may include lane markings identified based on the analysis of the at least one image. For example, the processing unit 110 may use information derived from at least one image to perform an analysis to identify one or more lane markings. Using the identified lane markings, the processing unit 110 may determine a correspondence between the detected traffic light and the lane in which the vehicle 200 is currently traveling.

[0289] At step 2330, processing unit 110 (e.g., via association determination module 2202) may use at least one indicator of the vehicle location, as determined at step 2320, to determine an association of each of the plurality of traffic light poles with vehicle 200. In some embodiments, processing unit 110 may rank the associations of the identified traffic light poles in at least one acquired image. The traffic light pole with the highest association ranking may be determined to be the associated traffic light pole. For example, in Fig. 20 In the illustrated case, the field of view of the image capture device 122 includes the traffic light pole 2012 and the traffic light pole 2014, but by using one or more evaluations, the processing unit 110 can determine that the traffic light pole 2014 has a higher relevance ranking value. One method by which the processing unit 110 determines that the traffic light pole 2014 has a higher relevance ranking than the traffic light pole 2012 is by evaluating the distance of each of the plurality of traffic light poles relative to the vehicle 200a. For example, in Fig. 20 In the case described in FIG. 2 , traffic light pole 2014 is closer to traffic light pole 2012. Therefore, traffic light pole 2014 is more likely to be associated than traffic light pole 2012. Other methods of determining association are described above with reference to association determination module 2202.

[0290] In some embodiments, the determination of the association of each of the plurality of traffic light poles with the vehicle 200 may include a preliminary check to eliminate unlikely traffic light poles. For example, when at least one acquired image includes three closer traffic light poles and two farther traffic light poles, the two farther traffic light poles may be classified as unlikely to be associated with the vehicle 200. By eliminating unlikely traffic light poles and ranking the association of the identified subjects of the traffic light poles in the at least one acquired image, the processing unit 110 may save processing power. In some embodiments, the association ranking may change as the vehicle 200 approaches a hub. For example, the orientation of a traffic light pole may change from different points on the road, and thus, the distance from the hub may have an impact on the probability that a given traffic light pole is associated with the vehicle 200. Therefore, the association ranking may be associated with a confidence level, which may take into account factors such as the distance from the hub when evaluating the traffic light pole. Further, when the confidence level is below a certain predetermined threshold, the processing unit 110 may periodically or continuously update the association ranking of the traffic light poles.

[0291] In order to determine the traffic lights associated with the vehicle 200, the association determination module 2202 may also use navigation information previously stored within the system 1000 (such as stored within the memory 140 / 150). Based on the determined 3D position of the vehicle 200 and / or the image capture device 122-126 as discussed above, the association determination module 2202 may perform a registration between the navigation map data and the vehicle 200. Based on the determination of the relative distance measurement as described above, the association determination module 2202 may use the navigation map data to determine the point in 3D space where the vehicle 200 (via the braking system 230) should brake in order to stop at each detected traffic light pole at the hub. Based on the 3D registration results, the association determination module 2202 may use the navigation map data to determine the lane assignment for each traffic light detected at the intersection. The association determination module 2202 may then determine the lane assignment within the system 100 and the image processor 190 and then perform the registration.

[0292] At step 2340, the processing unit 110 (e.g., via the state determination module 2204) may determine the state of a traffic light included in at least one traffic light pole determined to be associated with the vehicle 200 based on at least one acquired image. In some cases, the associated traffic light pole may include multiple illuminated traffic lights, and the state of each traffic light may depend on the type of traffic light. In one embodiment, the state of a traffic light refers only to the color it indicates, such as green, yellow, and red. The system 100 may use various techniques to identify the state of a traffic light. For example, the system 100 may identify an area of ​​one or more images including a traffic light, and perform an analysis of pixels in the area to determine the color of the pixels. After analyzing at least a threshold number of pixels in the area (e.g., two pixels, ten pixels, twenty pixels, etc.), the system 100 may determine the color of the traffic light, for example, by finding the average value of the pixels in the area.

[0293] Additionally or alternatively, the state of the traffic light may include the direction indicated by the traffic light. In a typical traffic light, the green indication allows the vehicle 200 to proceed. However, sometimes this information alone is not sufficient to determine whether it is safe to travel in a certain direction (such as when the green light only authorizes a turn). One method of determining which direction a traffic light indicates includes evaluating a database that associates each traffic light with one or more directions. Another method includes identifying the type of traffic light from the image data and determining the direction indicated by the traffic light from the environmental context. For example, in Fig. 20 In the second case described in step 2340 with respect to vehicle 200b, traffic light pole 2024 may be determined as an associated traffic light pole, but traffic light pole 2024 includes two illuminated traffic lights 2024a and 2024b. Therefore, in step 2340, processing unit 110 may determine that the state of traffic light 2024a is a red light for continuing straight, and the state of traffic light 2024b is a green light for turning right. In some embodiments, the determination of the state of the traffic light includes one or more of the following steps: determining the position of the traffic light within the associated traffic lights; determining whether the traffic light is illuminated; determining the color of the traffic light; and determining whether the traffic light includes an arrow.

[0294] At step 2350, the processing unit 110 (e.g., via the system response module 2206) may generate a system response based on the determined state. In some embodiments, the processing unit 110 may generate one or more system responses (e.g., two or more responses), including different types of responses. One type of system response may include a navigation response, which may include, for example, starting driving, a change in acceleration, a change in speed, applying vehicle braking, terminating cruise control, and the like. For example, these system responses may include providing a control signal to one or more of the throttle system 220, the braking system 230, and the steering system 240 to navigate the vehicle 200. Another type of system response may include starting a timer and / or increasing a counter to provide statistical information about one or more driving processes to the driver of the vehicle 200. For example, the statistical information may indicate the number of times the vehicle 200 has encountered a red light during the driving process, and / or the duration of the driving process that the vehicle 200 spends waiting at the red light. Another type of system response may include providing various notifications (e.g., warnings and / or alarms) to the driver of the vehicle 200. Warnings and / or alerts may include, for example, notification of the color of an associated traffic light and / or the distance to a hub. The notification may be provided via a speaker or via an associated display (eg, touch screen 320).

[0295] Turn Lane Traffic Light Detection

[0296] The system 100 can detect that a vehicle, such as the vehicle 200, is traveling in a turning lane. A lane can refer to a designated or intended path of travel for a vehicle and can have marked boundaries (e.g., lines on a road) or unmarked boundaries (e.g., road curbs, road barriers, guardrails, parked vehicles, etc.) or constraints. For example, the system 100 can operate to make these detections and determinations based on visual information acquired via one or more image capture devices. In some embodiments, these detections and determinations can also be made at least in part with respect to map data and / or sensed vehicle positions. In addition to determining the state of the vehicle 200 in the turning lane, the system 100 can identify traffic lights associated with the lane and can be configured to determine the state of the traffic lights based on an analysis of the road environment information and the characteristics of the determined traffic lights.

[0297] Fig.24Vehicle 200 is illustrated traveling on road 800 in which the disclosed systems and methods for detecting objects on the road may be used. Vehicle 200 is depicted as being equipped with image capture devices 122 and 124, more or fewer cameras may be employed on any particular vehicle 200. As shown, road 800 may be subdivided into lanes, such as lanes 810 and 820. Lanes 810 and 820 are shown as examples; a given road 800 may have additional lanes based on the size and nature of the road. Fig.24 In the example of FIG. 8 , vehicle 200 is traveling in lane 820 and can be seen approaching intersection 2402 with crossroad 2404. Traffic on lane 820 at intersection 2402 is controlled by traffic light pole 2406. As discussed in detail below, system 100 can determine the state of traffic light pole 2406 and generate a system response that affects the operation of vehicle 200. Further illustrated on road 800 is warning line 2410 leading to stop line 2412.

[0298] The processing unit 110 may be configured to determine one or more lane constraints associated with each of the lanes 810 and 820, the intersection 2402, or the crossroad 2404 based on a plurality of images acquired by the one or more image capture devices 122-126 that the processing unit 110 may receive via the data interface 128. In accordance with some embodiments, the lane constraints may be identified by visible lane boundaries (such as dashed or solid lines marked on the road surface). Additionally or alternatively, the lane constraints may include a curb or obstacle of the road surface. Additionally or alternatively, the lane constraints may include markings (e.g., Botts points). In accordance with some embodiments, the processing unit 110 (via the lane positioning module 2520 described in detail below) may determine the constraints associated with the lanes 810 / 820, the intersection 2402, or the crossroad 2404 by identifying, for example, the midpoint of the overall road width of the road 800 or the crossroad 2404 or the midpoint of one of the lanes 810 / 820. When, for example, there are no lines drawn or marked designating road lanes such as lanes 810 / 820, the processing unit 110 may identify lane constraints in an alternative manner (such as by estimation or inference) based on known road parameters. The processing unit 110 may also determine the physical relative distances between various constraints and detected objects (such as traffic lights).

[0299] When building autonomous vehicles or red traffic light warning systems, distance estimation for hubs with traffic lights can be challenging because the location of the traffic light can be behind or in the middle of the hub. For example, determining the 3D location for each traffic light at an intersection can hint at the braking point for the vehicle, but the stop line can provide the exact location.

[0300] In the current system, the detection of the road 800, the crossroad 2404 or the intersection 2402 and the constraints and paths constituting the lane 810 / 820 may include a processing unit 110 that determines its 3D model via a camera coordinate system. For example, the 3D model of the lane 810 / 820 may be described by a cubic polynomial. In addition to the 3D modeling of the driving lane, the processing unit 110 may also perform multi-frame estimation of host motion parameters such as the speed, yaw and pitch rate and acceleration of the vehicle 200. The processing unit 110 may also determine a road elevation model to convert information obtained from multiple images into a three-dimensional space. A feature of the current system is that an overall model for stationary objects at a traffic intersection can be created simultaneously, rather than creating a separate model for each detected object. Therefore, the system 100 may be configured to determine the absolute distance for an object or lane constraint at a distance, for example, within about one hundred meters. In some cases, the system 100 may be configured to determine the absolute distance for an object or lane constraint at other distances (e.g., within 125 meters, within 150 meters, etc.). Further, the system 100 may be configured to determine which traffic light pole (eg, traffic light pole 2406 ) is associated with the vehicle 200 within a distance of approximately eighty meters from the stop line 2412 .

[0301] Accurate distance estimation for traffic lights at intersections can be achieved through a process known as expansion and scaling. The expansion process can use relative distances to determine the distance from the vehicle 200 to an object such as the stop line 2412. For example, if two traffic light poles are detected in the image data, and the distance between the two traffic light poles increases by 5% when the vehicle moves a distance of ten meters, the system calculates the distance to be 200 meters.

[0302] Fig.25 1 is an exemplary block diagram of memory 140 and / or 150 consistent with the disclosed embodiments, which memory 140 and / or 150 may store instructions for performing one or more operations. Fig.25 As illustrated in , the memory 140 / 150 may store one or more modules for performing object detection and response described herein. For example, the memory 140 / 150 may store a traffic light detection module 2510, a lane positioning module 2520, and an action module 2530. The application processor 180 and / or the image processor 190 may execute instructions stored in any module 2510-2530 (included in the memory 140 / 150). It will be understood by those skilled in the art that references to the processing unit 110 discussed below may refer to the application processor 180 and the image processor 190 individually or collectively. Therefore, any of the following processing steps may be performed by one or more processing devices.

[0303] Traffic light detection module 2510 may store instructions that, when executed by processing unit 110, may detect the presence and status of a traffic light pole (such as traffic light pole 2406). Fig.26A , Fig.26B and Fig. 27 As discussed, the traffic light detection module 2510, in conjunction with the image processor 190, can perform image processing of one or more images acquired by the one or more image capture devices 122-126. The traffic light detection module 2510 can also determine the state of the traffic light pole 2406, including determining whether a light within the traffic light pole 2406 includes an arrow. The traffic light detection module 2510 can determine other information associated with the state of the traffic light pole 2406, including, without limitation, whether any traffic lights associated within the traffic light pole 2406 are illuminated (in a fixed or flashing manner), determine the position of the traffic lights within the traffic light pole (i.e., the horizontal orientation versus the vertical orientation of the traffic lights), or determine the color associated with the traffic lights. In some embodiments, the traffic light detection module 2510 can store information determined for a particular direction of travel, a particular intersection 2402, and / or a particular traffic light pole 2406 within the processing unit 110 (such as in the memory 140 / 150). In these embodiments, when the vehicle 200 returns to the same intersection, the previously determined and saved information may be used in the future.

[0304] In some embodiments, a flashing traffic light may be determined by image analysis of multiple images acquired at a predetermined or known capture rate (e.g., images acquired at 1 second, 1.5 seconds, 2 seconds, etc. are analyzed separately). For example, the system 100 may analyze the image data to identify a lighting pattern among the multiple images. The system 100 may also determine an area of ​​the captured image that is determined to be within the boundaries of a particular traffic light of interest in the traffic light pole. The system 100 may then determine the color of the traffic light by analyzing the pixels in the area determined to be within the boundaries of the traffic light of interest.

[0305] The lane positioning module 2520 may store instructions that, when executed by the processing unit 110, may assist the system 100 in determining the location of the vehicle 200. In some embodiments, the step of determining the location of the vehicle 200 may include determining at least one indicator of the vehicle's location via visual determination or by analysis of at least one image received from the image capture device 122-126 via the data interface 128. In these embodiments, the at least one indicator of the vehicle's location may include a distance from the vehicle to one or more lane constraints or lane markings associated with the current lane in which the vehicle is traveling, or to markings associated with an intersection, such as a warning line 2410 and a stop line 2412. The distance from the vehicle to the location or object may be determined based on one or more of, for example, image data, GPS information, or analysis of data from a location sensor. Further, the at least one indicator of the vehicle's location may include an arrow associated with a traffic light pole having at least one associated traffic light. Additionally or alternatively, the at least one indicator of the vehicle's location may include a vehicle location acquired via a GPS or similar coordinate system. In some embodiments, the lane location module 2520 may store lane constraint information determined for a particular road 800 and its lanes 810 / 820. For example, the processing unit 110 may store the information in the memory 140 / 150. In these embodiments, when the vehicle 200 returns to the same intersection, the previously determined and saved information may be used in the future. For example, GPS information may be used to determine that the vehicle 200 has returned to the same intersection.

[0306] Consistent with disclosed embodiments, lane location module 2520 may use at least one indicator from the vehicle position to determine whether a system response is required or recommended to change the operation of vehicle 200. Additionally or alternatively, lane location module 2520 may receive information from other modules (including position sensor 130) or other systems indicating the presence of other features in the image data (such as other vehicles, curvature of the road, etc.).

[0307] The action module 2530 may store instructions that, when executed by the processing unit 110, may assist the system 100 in performing one or more actions relative to the operation of the vehicle 200 based on information received from one or more sources (such as the position sensor 130, the image processor 190, the traffic light detection module 2510). In some embodiments, the action module 2530 may receive information (from, for example, the traffic light detection module 2510) regarding the state of a traffic light at an intersection (such as the traffic light pole 2406 at the intersection 2402 discussed above). Other information received by the action module 2530 may include a determination of whether the vehicle is in a turn lane and whether the traffic light pole 2406 includes an arrow.

[0308] Based on this received information, the action module 2530 can then generate a system response that affects the operating state of the vehicle 200, such as causing the system 100 to provide control signals to one or more of the throttle system 220, the braking system 230, and the steering system 240 to navigate the vehicle 200 (e.g., by accelerating, turning, etc.). For example, in some embodiments, the system 100 can determine that the turning lane traffic light authorizes the vehicle 200 to make a turn. In these embodiments, the action module 2530 can send an instruction to the steering system 240, and the steering system 240 can execute the instruction to turn the vehicle 200 through the intersection 2402 into a new lane of travel associated with the crossroad 2404. In other embodiments, the system response initiated and executed by the action module 2530 can include any or all of the following: providing a visual or audible notification to the operator of the vehicle, applying the vehicle brakes, suspending the cruise control function, or initiating one or more automatic turning maneuvers.

[0309] Additionally or alternatively, the action module 300 can send instructions to other systems associated with the vehicle 200, such as the brake system 230, the turn signal, the throttle system 220, etc. In some embodiments, the action module 2530 can instead provide a human operator of the vehicle with audio, visual, or tactile feedback representative of information collected from associated systems and / or sensors. The human operator can then act on the feedback to cause the vehicle to turn.

[0310] Fig.26A Provided Fig.24 . Vehicle 200 is again traveling in lane 820 of road 800 and is approaching intersection 2402 with crossroad 2404. Vehicle 200 is again equipped with image capture devices 122 and 124; more or fewer devices may be associated with any particular vehicle 200. For simplicity of illustration, Fig.26A The road 800 in FIG. 2 is depicted as a one-way street oriented from top to bottom on the page with two travel lanes 810 and 820. The crossroad 2404 is a two-way street oriented left to right across the entire page with one travel lane going in each direction.

[0311] At the intersection 2402, to the left of the road 800, is a traffic light pole 2406. A lane arrow 2408 is drawn on the surface of the lane 820 in front of the vehicle 200, indicating that the lane 820 is a left turn lane. Also drawn or attached on the surface of the road 800 is a warning line 2410 leading to a stop line 2412.

[0312] Consistent with the disclosed embodiments, the system 100 may be configured to determine whether the driving lane (here, lane 820) of the vehicle 200 approaching the intersection is a turning lane; determine whether a traffic light pole (here, traffic light pole 2406) controls the intersection; determine the state of the traffic lights in the traffic light pole; and determine whether the traffic lights include arrows. Fig. 27 and process 2700 to describe the process in more detail.

[0313] In brief, the system 100 associated with the vehicle 200 can determine the position of the vehicle 200 inside the road 800 via one or more of the processing unit 110, the position sensor 130, the traffic light detection module 2510, or the lane positioning module 2520. Additionally or alternatively, the system 100 can collect information from at least one indicator of the vehicle's position. As discussed above in conjunction with the lane positioning module 2520, at least one indicator of the vehicle's position can include a distance from the vehicle 200 to one or more lane constraints or lane markings associated with the current lane (such as lane 820) in which the vehicle is traveling, or to markings associated with the intersection 2401 (such as warning lines 2410 and stop lines 2412). Further, at least one indicator of the vehicle's position can include an arrow associated with the traffic light pole 2406. Additionally or alternatively, at least one indicator of the vehicle's position can include a vehicle position acquired by GPS or a similar coordinate system.

[0314] exist Fig.26AIn the example of , there are multiple indicators of the location of the vehicle 200. One or more image capture devices 122-126 associated with the vehicle 200 may be configured to capture multiple images of the area in front of the vehicle 200 that may help determine the location of the vehicle 200. For example, the image may include a lane arrow 2408 indicating that the lane 820 is a left turn lane of the intersection 2402. The image may also include a traffic light frame 2406, and may indicate that one or more of the individual lights inside the traffic light frame 2406 include arrows indicating the turning lane condition. One or more image capture devices 122-126 may also capture images of warning lines 2410 or stop lines 2412 associated with the road 800 or the intersection 2404. Further, the system 100, via the lane positioning module 2520, may determine the location of the vehicle 200 inside the turning lane by measuring the distance s1 from the vehicle 200 to the stop line 2412. Still further, the system 100 may perform optical character recognition (OCR) processing to obtain text (e.g., text from signs and / or road markings) included in one or more of the captured images. The system 100 may then use the text information as part of or as a basis for determining whether the vehicle 200 is inside a turn lane. For example, the system 100 may recognize specific words that indicate a turn lane (e.g., "turn right," "turn left," etc.).

[0315] The measurements may be direct measurements (such as via position sensor 130), or may be determined by image processor 190 via analysis of captured image data and may be used in conjunction with map data. The measurements may be measured from any portion of the interior or exterior vehicle 200, including, but not limited to: the front of the vehicle 200, portions of the vehicle 200 such as headlights or front license plates, mounted locations of image capture devices 122-126, a determined centroid of the vehicle 200, the rear of the vehicle 200, one or more windshields or reflectors associated with the vehicle 200, wheels of the vehicle 200, the right or left side or windows of the vehicle 200, points associated with the roof of the vehicle 200, or points associated with the chassis of the vehicle 200.

[0316] In some embodiments, determining the distance from the vehicle 200 to the traffic light pole 2406 may be sufficient to assist the processing unit 110 in calculating the braking distance or other such measurement. However, in other embodiments, the traffic light pole 2406 may be positioned to pass the point at which the vehicle 200 would be required to stop. In these embodiments, the distance to the intersection may be determined by using one or more of the warning line 2410 or the stop line 2412.

[0317] As above combined Fig.24As discussed, these distances can be calculated using an estimation process that employs expansion and scaling. At an intersection 2402 having two traffic light poles, for example, the system 100 can measure the relative distance between the two poles as captured in the image data for a period of time, and then use the relative distance to estimate the distance to the stop line 2412. In some embodiments, these measurements can be repeated over time to improve accuracy. Accuracy can also be improved by other methods; for example, if there are three or four traffic light poles located at the intersection 2402, the relative distance between each of the poles can be calculated and averaged. Additionally or alternatively, as discussed above, a Kalman filter can be used to estimate the distance.

[0318] The system 100 still requires additional information or input to determine the location of the stop line 2412 relative to the intersection 2402 and / or the traffic light pole 2406. In some embodiments, the system 100 can use previously stored map data collected from previous trips to the intersection. This information can be received from the traffic light detection module 2510 or the lane positioning module 2520. The distance measurement value Z can also be derived from the image data using the following formula:

[0319] Z=fW / w

[0320] Where W is the known distance between the two traffic light poles 2406, w is the distance in the image data in pixels, and f is the focal length of the particular image capture device 122-126 in pixels.

[0321] exist Fig.26A In the example, illustration 2602 shows two states of a traffic light pole 2406 associated with the example. Based on image data from at least one image received from image capture device 122-126 via data interface 128, system 100 can be configured to determine the state of traffic light pole 2406. Fig. 27 This determination is described in more detail in 2700 and 2710. When approaching intersection 2402, system 100 may first determine that traffic light pole 2406 is displaying a fixed red light, as seen in image 1 of inset 2602. Fig.26AIn the example of , lane 820 is a left turn lane, so unless road 800 is located in a jurisdiction where left turns on red are legal (or if intersection 2404 is configured on the page as a one-way street with two lanes flowing from right to left), vehicle 200 must stop at stop line 2412 and remain stationary until it is determined that traffic light gantry 2406 has changed. In an alternative embodiment (not shown), vehicle 200 can travel in a lane determined to be a right turn lane, and assuming that right turns on red are legally allowed in the jurisdiction, system 100 can generate a system response that enables vehicle 200 to turn right while traffic light gantry 2406 remains red.

[0322] Subsequently, as seen in Image 2 of illustration 2602, the traffic light pole 2406 changes from a fixed red light to a fixed green arrow indicating a protected left turn. The system 100 can be configured to detect not only that the color of the light has changed, but also that an arrow has appeared inside the light based on pixel-level image analysis of the acquired image of the traffic light pole 2406 by the image processor 190. In some embodiments, in order for the system 100 to detect that the color of the traffic light has changed, the system 100 can analyze the captured images for a certain period of time (e.g., 1 second, 5 seconds, 10 seconds, 15 seconds, 20 seconds, 30 seconds, 1 minute, etc.) to identify the change in state of the traffic light. The system 100 can also determine at one time that the traffic light pole 2406 changes to display a green arrow whether it is safe or authorized to proceed in the direction indicated by the state of the light. Even if the light can display a green arrow, various circumstances (not shown) may require the vehicle 200 to delay further before completing the turn. For example, a pedestrian may be crossing the intersection 2404. Although in Fig.26A The middle road 800 is a one-way street, but in alternative embodiments (such as described below and in Fig.26B ), the road may be a two-way street, and the vehicle 200 may also be aware of whether a second vehicle coming from the opposite direction is making a right turn from the other direction onto the intersection 2404 at a red light. Further, other obstacles may prevent a timely left turn; for example, a set of railroad tracks may be oriented parallel to the road 800 to its left, and although the left turn green arrow is illuminated, obstacles associated with the train tracks may fall down to allow the train to pass. These examples are not intended to be limiting. Once the system 100 determines that the conditions at the traffic light gantry 2406 and the intersection both allow a safe turn, the system 100 may generate a system response involving one or more of the throttle system 220, the braking system 230, the steering system 240, the turn signal, and / or other subsystems and modules that cause the vehicle 200 to automatically turn left, via the action module 2530.

[0323] Fig.26BOther intersection scenarios consistent with the disclosed embodiments are illustrated. Fig.26B In the example of FIG. 8 , a vehicle 200 (here represented as vehicle 200e) is traveling in lane 810 of road 800 and is approaching intersection 2402 with crossroad 2404. Vehicle 200 is again equipped with image capture devices 122 and 124; more or fewer devices may be associated with any particular vehicle 200. Fig.26A Unlike the example of Fig.26B 2404 is a two-way street, and traffic can proceed in both directions. Intersection 2404 is still a left-right oriented two-way street that passes through the entire page with one lane of travel going forward in each direction.

[0324] At intersection 2402, to the right of road 800, is traffic light pole 2604. Lane arrow 2408 is drawn on the surface of lane 810 in front of vehicle 200, indicating that lane 820 is a left turn lane. As before, consistent with the disclosed embodiments, system 100 can be configured to determine whether the lane of travel of vehicle 200 approaching the intersection (this time, lane 810) is a turn lane; determine whether a traffic light pole (here, traffic light pole 2604) controls the intersection; determine the state of the traffic light; and determine whether the traffic light includes an arrow.

[0325] The system 100 may operate in substantially the same manner as described above to make these determinations. Fig.26B In the example of FIG. 2402, illustration 2606 demonstrates that traffic light pole 2604 is displaying a flashing or blinking yellow light as vehicle 200e approaches intersection 2402. The non-stationary nature is indicated by the dashed line of the arrow in illustration 2606. As may be familiar to one of ordinary skill in the relevant art, some jurisdictions use flashing yellow lights during certain times of the day to essentially function as a "yield" or "stop" sign, while still allowing a lane of travel to come to a complete stop with a red light or fully authorize a lane of travel to proceed with a green light.

[0326] As outlined above, system 100 may determine that traffic light pole 2604 is displaying a flashing yellow light, and may also determine that lane 810 is a turn lane. Fig.26B In the case described in FIG. 24, the system 100 may be required to make other determinations in order for the vehicle 200e to successfully cross the intersection 2402. As will be appreciated by those skilled in the art, the flashing yellow light authorizes approaching vehicles to turn at will, as long as it is safe to do so. It is expected that the driver will be cautious at the intersection, and the driver determines that turning too early will not create a potential collision or other danger.

[0327] One such determination that the system 100 may make is whether one or more vehicles are approaching the intersection from another direction. For example, while a travel lane traveling in one direction of the road may have a traffic light with a flashing yellow light in the turn lane, traffic traveling in the opposite direction may have a green light to proceed straight through the intersection. Therefore, if the operator of the vehicle does not yield at the flashing yellow light signal, a collision may occur. Fig.26B In the example of , the second vehicle 200f is indeed traveling straight through the intersection 2402. The system 100 can be configured to detect the presence of the vehicle 200f via image data captured from one or more image capture devices 122-126, and determine the location of the vehicle within the road / intersection and its speed using one or more of the position sensor 130, the image processor 190, the traffic light detection module 2510, or the lane positioning module 2520. Upon detecting and analyzing the vehicle 200f, the system 100 (via the action module 2530) can cause the braking system 230 to remain engaged until the vehicle 200f has passed.

[0328] Once no other vehicles are approaching the intersection from the opposite direction, the system 100 may generate a system response based on the determination via the action module 2530. The system response may include one or more of providing feedback to the operator of the vehicle 200, applying vehicle brakes via the braking system 230, suspending cruise control, or initiating an automatic turning maneuver onto the intersection 2404 using one or more of the throttle system 220, the braking system 230, or the steering system 240. First, however, the system 100 may analyze the image data for the presence of any other obstacles or objects associated with the intersection 2402 or the intersection 2404. For example, in Fig.26B In the example of FIG. 8 , pedestrian 2608 can be seen walking across intersection 2404 because another lane of road 800 has a green light in that direction. System 100 can detect the presence of pedestrian 2608 via image processor 190 by analyzing one or more of the multiple images captured by one or more image capture devices 122-126. As described above, other subsystems and modules of system 100 can cooperate to detect the speed of pedestrian 2608 and can notify system 100 when pedestrian 2608 is no longer on the road and it is safe to turn.

[0329] Fig. 27 A process 2700 for detecting the state of a traffic light consistent with the disclosed embodiments is illustrated. The steps of the process 2700 may be performed by one or more of the processing unit 110, the image acquisition unit 120, the position sensor 130, the image processor 190, the traffic light detection module 2510, the lane positioning module 2520, or the action module 2530.

[0330] At step 2710, the process 2700 may include acquiring a plurality of images of an area in front of the vehicle 200 using at least one image capture device 122, 124, and / or 126. The area may include one or more traffic light poles (such as the traffic light poles 2406 and 2604 described above) having at least one traffic light. The processing unit 110 may receive the plurality of images from the image capture device(s) via the data interface 128. The plurality of images may then be processed in real time by the image processor 190 of the processing unit 110. The image processor 190 may process at least one image captured by the at least one image capture device, as well as other images captured at a later point in time.

[0331] At step 2720, the process 2700 may determine whether the vehicle 200 is traveling in a turning lane based on at least one indicator of the vehicle's position. As generally described above, various indicators of position may be used to determine whether the driving lane is a turning lane. In some embodiments, the position indication may include a vehicle position obtained by GPS. In other embodiments, the system 100 may identify lane markings or constraints (such as a lane arrow 2408 indicating that the lane is a turning lane). Alternatively, the image processor 190 may determine that the arrow is associated with a traffic light pole that controls lane travel. In addition, various elements of the system 100 may be used to measure distances. For example, the position sensor 130 and / or the lane positioning module 2520 may send information about the relative position of the vehicle 200 within the lane 810 / 810, including how far the vehicle 200 is from the intersection 2402 (the distance s1 described above), to the processing unit 110. The end of the lane may be determined based on a measured distance or visually (eg, by analyzing one or more images acquired by one or more image capture devices 122 - 126 and detecting the presence of warning line 2410 or stop line 2412 ).

[0332] At step 2730, the process 2700 may perform image processing of at least one image received via the data interface 128 to determine whether an arrow is present in the image data associated with the traffic light. The traffic light detection module 2510 and / or the image processor 190 may be configured to analyze pixels of the image corresponding to the positions of the respective traffic lights within the traffic light pole. One or more image processing algorithms may be applied to the pixels to determine whether a fixed or blinking / flashing arrow is present in the associated area. Figures 28 to 30B Let's discuss further examples of how the system 100 may analyze one or more images to determine whether a traffic light displays an arrow.

[0333] At step 2740, the process 2700 may determine the state of the traffic light. As discussed above, the step of determining the state of the traffic light may include, without limitation, determining whether the traffic light is illuminated (and / or functional), determining the location of an illuminated traffic light within a traffic light frame, or determining a color associated with the traffic light (i.e., red, yellow, or green). The list is for example only, and any information associated with the traffic light may be determined and processed by the system 100 and the processing unit 110. For example, the system may be configured to determine whether the traffic light is fixed or flashing based on image processing of a series of acquired images. The image processor 190 may determine that in these embodiments, the same traffic light within the frame is illuminated in one image, then dimmed in sequence, and then immediately illuminated again; this would indicate that the particular light is flashing. In some embodiments, the system 100 may be configured to identify traffic light conditions that are particular to a particular jurisdiction in which the vehicle is traveling. This information may be pre-programmed into the system 100, such as in data (not shown) stored within the memory 140 / 150.

[0334] At step 2750, the process 2700 may generate a system response via the system 100 that affects the operation of the vehicle 200. The response initiated by the system 100 may be based on any or all of the following determinations: determining the state of the traffic light, determining whether the traffic light includes an arrow, and determining whether the vehicle is in a turning lane. As discussed above, the specific actions initiated by the system 100 via the action module 2530 may include, without limitation: providing a visual or auditory feedback notification to the operator of the vehicle 200 regarding the traffic light; applying vehicle brakes via the braking system 230; suspending the cruise control function; or using one or more of the throttle system 220, the braking system 230, or the steering system 240 to proceed to initiate an automated turning maneuver and proceed through the intersection.

[0335] Additionally, in some embodiments, as part of the functionality of the system 100, the working module 2530 and / or the processing unit 110 may also be configured to determine whether the turning lane in which the vehicle is located is authorized to proceed in the direction indicated by the determined state of the traffic light. In other words, even if the traffic light indicates that a particular system response is legally permitted, the system 100 may provide an additional level of safety protection for the driver of the vehicle 200 by further determining whether such an action is actually possible and safe. For example, as described above in conjunction with Fig.26B As discussed, the system 100 and image processor 190 may be configured to detect the presence of other obstacles associated with a road or intersection (such as the presence of pedestrians or other vehicles on the street). In some embodiments, other vehicles may proceed normally through the intersection (such as in Fig.26BIn these embodiments, vehicle 200e could theoretically turn onto intersection 2404, but system 100 may also take into account the fact that vehicle 200f is proceeding through the intersection when the light is green. Alternatively, the second vehicle may appear in an unusual or illegal manner, such as if the operator of another vehicle ran a stop sign or red light, or if the vehicle has broken down at the intersection or on one of the roads.

[0336] Still in other embodiments, the step of determining whether the turning lane in which the vehicle is located is authorized to proceed in the direction indicated by the determined state of the traffic light may also include: determining whether the indicated state of the light is actually followed based on the vehicle's intended destination. In embodiments involving autonomous vehicles, it can be assumed that the operator and / or passenger has indicated their desired destination to the processing unit 110 of the system 100. Therefore, by using GPS positioning technology or other such information, the system 100 can be configured to determine whether the turn indicated by the traffic light is actually associated with the intended destination. For example, in some embodiments, the vehicle 200 has been unintentionally stalled in the turning lane, and proceeding to the intended destination may involve merging into an adjacent lane (with the help of the lane positioning module 2520) and continuing to proceed straight through the intersection. Therefore, the system 100 can be configured to determine whether to turn left, turn right, or continue straight through the intersection based on the intended destination information.

[0337] Traffic light detail detection

[0338] In some embodiments, the system 100 may provide additional functions related to the identification of traffic signals. For example, the system 100 may provide a traffic light detail detection function based on the analysis of the physical features of the traffic light to provide feedback. To implement such a function, the processing unit 110 may process the image captured by at least one of the image capture devices 122, 124, 126 and employ super-resolution techniques to identify specific features of the traffic light in the image, wherein the specific features may indicate useful information to the system 100. For example, the system 100 may identify a circular signal and an arrow-shaped signal of a traffic light that have been illuminated, and distinguish between them.

[0339] In some embodiments, the memory 140 and / or 150 may store instructions that are programmed to provide traffic light detail detection functionality when executed by the processing device. Fig.28As shown in , the memory 140 and / or 150 may store an image alignment module 2810, a pixel expansion module 2820, an image analysis module 2830, and a database 2840. The image alignment module 2810 may store instructions for aligning multiple images captured by the image capture devices 122, 124, and 126. The pixel expansion module 2820 may store instructions for increasing the resolution of at least a portion of the image to allow the image to be scaled to a standard size. The image analysis module 2830 may store instructions for analyzing the image to detect and identify features of the image. The database 2830 may be configured to store data associated with providing a traffic light detail detection function and provide the data when needed. Further, the image alignment module 2810, the pixel expansion module 2820, and the image analysis module 2830 may store instructions that can be executed by one or more processors (e.g., the processing unit 110) individually or in various combinations with each other. For example, the image alignment module 2810, the pixel expansion module 2820, and the image analysis module 2830 may be configured to interact with each other and / or other modules of the system 100 to perform functions consistent with the disclosed embodiments.

[0340] Database 2840 may include one or more storage devices that store information and are accessed and / or managed by a computing device (such as processing unit 110). Fig.28 As shown in , database 2840 can be located in memory 150. In other embodiments, database 2840 can be located remotely from memory 150 and accessible to other components of system 100 (e.g., processing unit 120) via one or more wireless connections (e.g., wireless networks). Although one database 2840 is shown, it should be understood that multiple separate and / or interconnected databases can make up database 2840. Database 630 can include a computing component (e.g., a database management system, a database server, etc.) that is configured to receive and process requests for data stored in a memory device associated with database 2840 and provide data from database 2840 (e.g., to processing unit 110).

[0341] In some embodiments, the database 2840 may be configured to store data associated with providing a traffic light detail detection function. For example, the database 2840 may store data (such as images, maps, algorithms, sets of values, etc.) that may allow the processing unit 110 to identify information detected in an image. For example, the database 2840 may store a set of average pixel values ​​corresponding to each of a circular signal and an arrow-shaped signal of a traffic light.

[0342] In some implementations, the system 100 can provide feedback based on details of a traffic light detected in one or more captured images. For example, the system 100 can be configured to differentiate between the color, shape, and position of signals associated with a traffic light in order to provide specific feedback based on the meaning of the detected details (e.g., an arrow-shaped signal controls turning movement, while a circular signal controls straight ahead movement).

[0343] Fig.29A and Fig.29B Exemplary objects that may be detected by system 100 are described. Fig.29A A traffic light pole 2900A including a circular signal 2910 is depicted. Fig.29B A traffic light pole including an arrow-shaped signal 2920 is depicted. While both objects include a traffic light, they include different details indicating different messages. In an exemplary embodiment, the system 100 may be configured to analyze pixels of one or more images to identify distinguishing features, such as the circular signal 2910 and the arrow-shaped signal 2920.

[0344] Fig.29C An exemplary portion of an image 2915 of a circular signal 2910 is depicted, including a plurality of pixels 2930. The system 100 can be configured to analyze the pixels 2930 to identify objects and / or details found in the image. For example, the image analysis module 2830 can be configured to identify features (such as brightness, color, contrast, hue, transparency, etc.) associated with each pixel 2930, and determine objects and / or details associated with the image based on the pixel analysis. Each feature identified by the image analysis module 2830 can be given a value. For example, the image analysis module 2830 can assign a value of "one" to a pixel identified as the color red, and assign a value of "zero" to any other pixel.

[0345] In some embodiments, system 100 can be configured to use super-resolution techniques to allow for more accurate analysis of objects in multiple images. For example, pixel expansion module 2820 can modify image 2915 to divide one or more of pixels 2930 into subsets of pixels 2940. Fig.29C 2940, but it should be understood that any portion or all of the pixels 2930 may be divided in this manner. Thus, the pixel expansion module 2820 may increase the number of pixels associated with a particular portion of the image.

[0346] In an exemplary embodiment, the pixel expansion module 2820 can increase the number of pixels associated with a particular object in an image so that the portion of the image including the object includes a selected number of pixels (e.g., a standard number of pixels). In this way, even if multiple images include the same object of different sizes (e.g., because the images were taken at different distances from the object), the images can be modified so that the portion of the image including the object includes the same number of pixels, which can be directly compared and averaged to more reliably determine the characteristics (e.g., color) of the portion of the object located inside the pixels.

[0347] Scaling the selected portion of the images so that the same number of pixels are associated with the object in each image can allow the images to be directly compared (e.g., comparing the value of one pixel in an image to the same pixel in another image). In this way, multiple images can be considered and analyzed by the image analysis module 2830, thereby increasing accuracy compared to the analysis of one image. For example, super-resolution techniques can increase the likelihood that the image analysis module 2830 can correctly distinguish between a circular signal 2920 and an arrow-shaped signal 2930 in an image.

[0348] Fig.29D An exemplary portion of image 2925 of arrow-shaped signal 2920 is depicted. As described above with respect to image 2915, image 2925 may be divided into pixels 2930, and each pixel 2930 may be expanded into a subset of pixels 2940. Fig.29C and Fig.29D , pixels 2930 differ around the edges of the respective signals 2920, 2930. The image analysis module 2830 may be configured to identify one or more characteristics of the pixel 2930 to determine whether the pixel corresponds to a circular signal 2910 or an arrow-shaped signal 2920. The expansion of the pixel 2930 into the pixel 2940 may allow multiple images to be considered and aligned, thereby increasing the reliability of the detection.

[0349] Fig. 30A 3000A is an example process for determining whether a traffic light includes an arrow consistent with disclosed embodiments. In an example embodiment, system 100 may perform process 3000A to determine whether a portion of a traffic light detected in a plurality of images includes an arrow-shaped signal. An arrow-shaped signal (or "arrow") may be a type of signal for a traffic light in which a visual component of the signal takes the shape of an arrow. An arrow may be a signal indicating that a vehicle may be allowed to turn (e.g., turn in the direction of the arrow) at this time.

[0350] At step 3002, the processing unit 110 may receive a plurality of images of a traffic light. For example, one or more image capture devices 122, 124, 126 may acquire a plurality of images of an area in front of the vehicle 200. In some embodiments, such as when the vehicle 200 is approaching a hub, the area in front of the vehicle 200 may...

Claims

1. A control system for a vehicle, the control system comprising: a forward-facing camera that captures a plurality of images of a road in front of the vehicle; as well as A processing device, the processing device being configured to perform the following steps: providing feedback to a vehicle operator of the vehicle to change lanes to a new lane in which the vehicle is not traveling based on a termination of a current lane indicated by a first traffic sign identified in the plurality of images; updating a distance from the vehicle to a second traffic sign based on the position of the vehicle, the second traffic sign used to constrain vehicle operation to the new lane; as well as After the vehicle changes to the new lane, communicate with the steering system to control the vehicle to be relatively centered in the new lane based on the distance from the vehicle to the second traffic sign indicating the lane boundary, whereby the distance is used to steer the vehicle to a safe operating position.

2. The control system according to claim 1, wherein: The feedback notifies the vehicle operator that the current lane is about to end.

3. The control system according to claim 1, wherein: The feedback includes a visual display.

4. The control system according to claim 1, wherein: The feedback includes a tactile response to alert the vehicle operator to steer into the new lane.

5. The control system according to claim 1, wherein: The processing device is configured to interact with the steering system to control the vehicle while changing lanes to the new lane.

6. The control system according to claim 1, wherein: The processing device is configured to interact with the steering system to control the vehicle after a lane change and maintain a vehicle trajectory in the new lane.

7. The control system according to claim 1, wherein: The camera is a stereo camera.

8. A control system for a vehicle, the control system comprising: a forward-facing camera that captures a plurality of images of a road in front of the vehicle; a steering assist system that provides steering assistance to an operator of the vehicle to change to a new lane based on the updated control signal from the image analysis system and updates a distance from the vehicle to a second traffic sign based on the position of the vehicle; as well as an interface that interacts with the steering assist system to assist the operator in changing to the new lane based on a control signal from the image analysis system, wherein the image analysis system provides an end of a current lane indicated by a first traffic sign that is identified in a plurality of images obtained by the image analysis system, Wherein, the steering assist system is further configured to control the vehicle to travel relatively centered in the new lane based on the distance from the vehicle to the second traffic sign indicating the lane boundary after the vehicle changes to the new lane, whereby the distance is used to steer the vehicle to a safe operating position.

9. The control system according to claim 8, wherein: The camera is a stereo camera.

10. An image analysis system, comprising: an image processing unit that identifies a first traffic sign in a plurality of images received from a camera of a vehicle, the first traffic sign indicating an end of a current lane in which the vehicle is traveling; as well as A signal generating system, the signal generating system generating a control signal to change to a new lane based on the termination of the current lane, wherein the signal generating system will generate a second control signal to remain in the new lane and update the distance from the vehicle to a second traffic sign based on the position of the vehicle, the second traffic sign being used to constrain vehicle operation to the new lane, and wherein the signal generating system will generate a third control signal to control the vehicle to travel relatively centered in the new lane after the vehicle changes to the new lane based on the distance from the vehicle to the second traffic sign indicating a lane boundary, whereby the distance is used to steer the vehicle to a safe operating position.

Images