Systems for navigating lane merges and lane splits

Abstract

Device comprising a processor of a host vehicle (200), wherein the processor is configured to cause the host vehicle to do the following: Received (902) from an image recording device (122), a plurality of images representative of an environment (1000) of the host vehicle (200), wherein the environment includes a road on which the host vehicle is traveling; Analyze (904) at least one of the multitude of images to identify a target vehicle (802) traveling in a lane of the road that is different from a lane in which the host vehicle (200) is traveling; Analyze (906) at least one of the plurality of images to identify at least one lane marker (1004A, 1004B) associated with the lane on which the target vehicle is traveling; Determine (912) at least one feature of the target vehicle (800) based on the plurality of images, wherein the at least one determined feature of the target vehicle (800) comprises a detected lateral movement of the target vehicle in relation to the at least one identified lane marking (1004A, 1004B); and Determine (914) a navigation action for the host vehicle (200) based on the determined feature of the target vehicle (800).

Description

STATE OF THE ART EP-A1-2 455 266 US-A1-2005 015 203 US-A1-2014 236 414

[0001] ARMINGOL ET AL: “IVVI: Intelligent vehicle based on visual information”, ROBOTICS AND AUTONOMOUS SYSTEMS, ELSEVIER SCIENCE PUBLISHERS, AMSTERDAM, NL, vol. 55, no. 12, 5 November 2007 (2007-11-05), pages 904-916, XP022328739, ISSN: 0921-8890, DOI: 10.1016 / J.ROBOT.2007.09.004 TECHNICAL FIELD

[0002] The present disclosure generally relates to autonomous navigation of vehicles. Furthermore, this disclosure relates to systems and methods for identifying lane markers, determining lane marker types, and navigating lane merges and lane splits. BACKGROUND INFORMATION

[0003] As technology continues to advance, the goal of a fully autonomous vehicle capable of navigating the roads is becoming more and more realistic. Autonomous vehicles may need to consider a variety of factors and make appropriate decisions based on these factors in order to reach an intended destination safely and accurately. For example, an autonomous vehicle must process and interpret visual information (e.g., information acquired by a camera), information from radar or lidar, and may also use information obtained from other sources (e.g., a GPS device, a speed sensor, an accelerometer, a suspension sensor, etc.). At the same time, to navigate to a destination, an autonomous vehicle may also need to identify its location on a specific roadway (e.g.,A vehicle may also need to navigate a lane (e.g., a specific lane on a multi-lane road), pass other vehicles, avoid obstacles and pedestrians, observe traffic signals and signs, navigate at appropriate intersections or road junctions, and respond to any other situation that arises or develops during the vehicle's operation. Furthermore, an autonomous vehicle may also need to consider road characteristics, such as different types of lane markings.

[0004] During navigation, an autonomous vehicle may typically travel on a road that includes different types of lanes. For example, the lane in which an autonomous vehicle is traveling may merge with an adjacent lane. In some cases, another vehicle within range of the autonomous vehicle may be traveling in an adjacent lane that is just ending and may enter in front of the autonomous vehicle. As another example, the lane in which an autonomous vehicle is traveling may split to form an additional, adjacent lane, such as an exit lane. In some cases, another vehicle may change lanes in front of the autonomous vehicle or exit a road and is no longer positioned in front of the autonomous vehicle.In each of these situations, the autonomous vehicle must take these lane changes and possible maneuvers that neighboring vehicles might perform in response to the lane changes into account when navigating. Furthermore, the autonomous vehicle may need to adjust its navigation path or speed during these lane changes to ensure safe and precise navigation.

[0005] US 2014 / 0236414 A1 relates to a technique for detecting nearby aggressive drivers by monitoring the position of an observed vehicle relative to a lane marking or by monitoring the lateral velocity between the host vehicle and the observed vehicle and adjusting the driving modes of an autonomous vehicle accordingly. US 2005 / 0015203 A1 relates to a vehicle-mounted lane change assistance or merging assistance system. Armingol, Jose Maria, et al., "IVVI: Intelligent vehicle based on visual information," Robotics and Autonomous Systems, Elsevier Science Publishers, Amsterdam, NL, Vol. 55, No. 12, November 5, 2007, pages 904-916 relates to visual perception modules for advanced driver assistance systems that include detection and classification of lanes (continuous, discontinuous, and merging) and road types. SUMMARY

[0006] According to aspects of the claimed invention, a computer-implemented method for navigating a host vehicle according to claim 1, a computer-readable medium for storing according to claim 14, and a system for navigating a host vehicle according to claim 15 are provided.

[0007] Embodiments consistent with the present disclosure provide systems and methods for autonomous navigation of vehicles. The disclosed embodiments may use cameras to provide features for autonomous navigation of vehicles. In accordance with the disclosed embodiments, the disclosed systems may, for example, include one, two, or more cameras monitoring the surroundings of a vehicle. The disclosed systems may determine one or more navigation actions for the vehicle, for example, based on an analysis of images captured by one or more cameras. The one or more navigation actions may also include other data, for example, global positioning system (GPS) data, sensor data (e.g., from an accelerometer, a speed sensor, a suspension sensor, etc.), and / or other map data.

[0008] In one embodiment, a system for navigating a host vehicle is disclosed. The system may include at least one processing device programmed to receive a plurality of images representing an environment of the host vehicle from an image capture device. The environment includes a road on which the host vehicle is traveling.The at least one processing device is further programmed to analyze at least one of the plurality of images to identify a target vehicle traveling in a lane of the road that is different from a lane in which the host vehicle is traveling; analyze at least one of the plurality of images to identify at least one lane marker associated with the lane in which the target vehicle is traveling; detect one or more lane marker characteristics of the at least one identified lane marker; use the one or more detected lane marker characteristics to determine a type of the at least one identified lane marker; determine at least one characteristic of the target vehicle; and determine a navigation action for the host vehicle based on the determined lane marker type and the determined characteristic of the target vehicle.

[0009] In another embodiment, a system for navigating a host vehicle is disclosed. The system may include at least one processing device programmed to receive a plurality of images representing an environment of the host vehicle from an image capture device. The environment includes a road on which the host vehicle is traveling.The at least one processing device is further programmed to analyze at least one of the plurality of images to identify a target vehicle traveling in a lane of the road on which the host vehicle is traveling; analyze at least one of the plurality of images to identify at least one lane marker associated with the lane; determine one or more characteristics of the at least one identified lane marker; use the at least one determined lane marker to determine a type of the at least one identified lane marker; determine at least one characteristic of the target vehicle; and determine a navigation action for the host vehicle based on the determined type of lane marker and the determined characteristic of the target vehicle.

[0010] In another embodiment, a method for navigating a host vehicle is disclosed. The method includes: receiving a plurality of images from an image capture device representative of an environment of the host vehicle, the environment including a road on which the host vehicle is traveling; analyzing at least one of the plurality of images, by a processing device, to identify a target vehicle traveling in a lane of the road that is different from a lane on which the host vehicle is traveling; analyzing at least one of the plurality of images to identify at least one lane marker associated with the lane on which the target vehicle is traveling; detecting one or more lane marker characteristics of the at least one detected lane marker;Using the one or more detected lane marker characteristics to determine a type of the at least one identified lane marker; determining at least one characteristic of the target vehicle; and determining a navigation action for the host vehicle based on the determined type of lane marker and the determined characteristic of the target vehicle.

[0011] In another embodiment, a method for navigating a host vehicle is disclosed. The method includes: receiving a plurality of images from an image capturing device representing an environment of the host vehicle, the environment including a road on which the host vehicle is traveling; analyzing at least one of the plurality of images by at least one processing device to identify a target vehicle traveling in a lane of the road on which the host vehicle is traveling; analyzing at least one of the plurality of images to identify at least one lane marker associated with the lane; detecting one or more characteristics of the at least one identified lane marker; using the at least one determined lane marker characteristics to determine a type of the at least one identified lane marker; determining at least one characteristic of the target vehicle;and determining a navigation action for the host vehicle based on the determined type of lane marking and the determined characteristic of the target vehicle;

[0012] In accordance with other disclosed methods, non-transferable computer-readable storage media may store program instructions executed by at least one processing device and performing any of the methods described herein.

[0013] The foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings: Fig. 1 is a schematic representation of an exemplary system consistent with the disclosed embodiments. Fig. 2A is a schematic side view of an exemplary vehicle including a system consistent with the disclosed embodiments. Fig. 2B is a schematic plan view of the Fig. 2A in accordance with the disclosed embodiments. Fig. 2C is a schematic plan view of another embodiment of a vehicle including a system consistent with the disclosed embodiments. Fig. 2D is a schematic plan view of another embodiment of a vehicle including a system consistent with the disclosed embodiments. Fig. 2E is a schematic plan view of another embodiment of a vehicle including a system consistent with the disclosed embodiments. Fig. 2F is a schematic diagram of exemplary vehicle control systems consistent with the disclosed embodiments. Fig. 3A is a schematic diagram of the interior of a vehicle including a rearview mirror and a user interface for a vehicle imaging system according to the disclosed embodiments. Fig. 3B is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and on the windshield of a vehicle in accordance with the disclosed embodiments. Fig. 3C an illustration of the Fig. 3B from a different perspective consistent with the illustrated embodiments. Fig. 3D is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and on the windshield of a vehicle in accordance with the disclosed embodiments. Fig. 4 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments. Fig. 5A is a flowchart illustrating an exemplary process for triggering one or more navigational responses based on monocular image analysis in accordance with disclosed embodiments. Fig. 5B is a flowchart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a set of images consistent with the disclosed embodiments. Fig. 5C is a flowchart illustrating an exemplary process for detecting road markings and / or lane geometry information in a set of images consistent with the disclosed embodiments. Fig. 5D is a flowchart illustrating an exemplary process for detecting traffic lights in a set of images consistent with the disclosed embodiments. Fig. 5E is a flowchart illustrating an example process for triggering one or more responses based on a vehicle path in accordance with the disclosed embodiments. Fig. 5F is a flowchart illustrating an exemplary process for determining whether a lead vehicle is changing lanes, in accordance with the disclosed embodiments. Fig. 6 is a flowchart illustrating an exemplary process for triggering one or more navigation responses based on stereo image analysis in accordance with the disclosed embodiments. Fig. 7 is a flowchart illustrating an exemplary method for triggering one or more navigation responses based on an analysis of three image sets, in accordance with the disclosed embodiments. Fig. 8A is an illustration of exemplary lane merges on a roadway in accordance with the disclosed embodiments. Fig. 8B is an illustration of exemplary lane separations on a roadway in accordance with the disclosed embodiments. Fig. 9 is a flowchart illustrating an exemplary process for determining one or more navigation actions based on lane change analysis in accordance with the disclosed embodiments. Fig. 10A is an illustration of an autonomous vehicle on a roadway having lane merging in accordance with the illustrated embodiments. Fig. 10B is an exemplary illustration of a lane change during lane merging in accordance with the disclosed embodiments. Fig. 11 is a flowchart illustrating another exemplary process for determining one or more navigation actions based on lane change analysis in accordance with the disclosed embodiments. Fig. 12A is an illustration of an autonomous vehicle on a roadway with lane separation consistent with the disclosed embodiments. Fig. 12B is an exemplary illustration of lane change analysis upon lane separation in accordance with the disclosed embodiments. Fig. 13 is a flowchart illustrating another example process or determination of one or more navigation actions based on lane change analysis in accordance with the disclosed embodiments. Fig. 14A-14C are illustrations of an exemplary lane change analysis for lane merging in accordance with the disclosed embodiments. Fig. 15A-15C are illustrations of an exemplary lane change analysis for a lane separation consistent with the disclosed embodiments. DETAILED DESCRIPTION

[0015] The following detailed description refers to the accompanying drawings. Wherever possible, the same numbers are used in the drawings and the following description to refer to the same or similar parts. While several illustrative embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, the components illustrated in the drawings may be replaced, added, or modified, and the methods illustrated herein may be modified by substituting, rearranging, removing, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the proper scope is defined by the appended claims. OVERVIEW OF AUTONOMOUS VEHICLES

[0016] As used in this disclosure, the term "autonomous vehicle" refers to a vehicle capable of implementing at least one navigation change without driver input. A "navigation change" refers to one or more changes to the vehicle's steering, braking, or acceleration / deceleration. To be autonomous, a vehicle does not have to be fully automatic (e.g., fully operational without a driver or without driver input). Rather, an autonomous vehicle includes those capable of operating under driver control during certain periods of time and without driver control during other periods. Autonomous vehicles may also include vehicles that control only some aspects of the vehicle's navigation, such as steering (e.g.,to maintain a vehicle's course between lanes) or perform some steering operations under certain circumstances (but not all circumstances) but leave other aspects to the driver (e.g., braking or deceleration under certain circumstances). In some cases, autonomous vehicles may take over some or all aspects of braking, speed control, and / or steering of the vehicle.

[0017] Since human drivers typically rely on visual cues and observations to steer a vehicle, transportation infrastructures are built accordingly, with lane markings, traffic signs, and traffic lights designed to provide visual information to drivers. Considering these design characteristics of transportation infrastructures, an autonomous vehicle may include a camera and a processing unit that evaluates visual information from the vehicle's surroundings. The visual information may, for example, include images depicting transportation infrastructure components (e.g., lane markings, traffic signs, traffic lights, etc.) observable by drivers and other obstacles (e.g., other vehicles, pedestrians, debris, etc.). Additionally, an autonomous vehicle may also utilize stored information, such as information that provides a model of the vehicle's surroundings when navigating.For example, the vehicle may use GPS data, sensor data (e.g., from an accelerometer, a speed sensor, a suspension sensor, etc.), and / or other map data to provide information about its surroundings while driving, and the vehicle (as well as other vehicles) may use this information to locate itself on the model. Some vehicles may also communicate with each other, exchange information, inform the other vehicle about hazards or changes in the vehicles' surroundings, etc. SYSTEM OVERVIEW

[0018] Fig. 1 is a block diagram representation of a system 100 in accordance with the exemplary disclosed embodiments. The system 100 may include various components depending on the requirements of a particular implementation. In some embodiments, the system 100 may include a processing unit 110, an image capture unit 120, a position sensor 130, one or more storage units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. The processing unit 110 may include one or more processing devices. In some embodiments, the processing unit 110 may include an application processor 180, an image processing processor 190, or other suitable processing device.Similarly, the image capture component 120 may include any number of image capture devices and components, depending on the requirements of a particular application. In some embodiments, the image capture unit 120 may include one or more image capture devices (e.g., cameras, CCDs, or any other type of image sensor), such as the image capture device 122, the image capture device 124, and the image capture device 126. The system 100 may also include a data interface 128 that communicatively connects the processing unit 110 to the image capture unit 120. For example, the data interface 128 may include one or more wired and / or wireless connections for transmitting image data captured by the image capture unit 120 to the processing unit 110.

[0019] The wireless transceiver 172 may include one or more devices configured to exchange transmissions over an air interface to one or more networks (e.g., cellular networks, the Internet, etc.) using a radio frequency, an infrared frequency, a magnetic field, or an electric field. The wireless transceiver 172 may use any known standard to transmit and / or receive data (e.g., Wi-Fi, Bluetooth®, Bluetooth Smart, 802.15.4, ZigBee, etc.). Such transmissions may include communications from the host vehicle to one or more remote servers. Such transmissions may also include (one- or two-way) communications between the host vehicle and one or more target vehicles in the vicinity of the host vehicle (e.g.,to facilitate the coordination of the host vehicle's navigation with respect to or in conjunction with target vehicles in the host vehicle's vicinity), or even a broadcast transmission to unspecified receivers in the vicinity of the transmitting vehicle.

[0020] Both the application processor 180 and the image processor 190 may include various types of hardware-based processing devices. For example, one or both of the application processor 180 and the image processor 190 may include a microprocessor, preprocessors (such as an image preprocessor), graphics processors, a central processing unit (CPU), support circuitry, digital signal processors, central processing units, memory, or any other type of device suitable for executing applications and for image processing and analysis. In some embodiments, the application processor 180 and / or the image processor 190 may include any type of single- or multi-core processor, mobile device microcontroller, central processing unit, etc. Various processing devices may be used, including, for example, processors from manufacturers such as Intel ®, AMDO, etc., which can span different architectures (e.g., x86 processor, ARMO, etc.).

[0021] In some embodiments, the application processor 180 and / or the image processor 190 may comprise any processor chip from the EyeQ series of Mobil-eye ® These processor designs each include multiple processor units with local memory and instruction sets. Such processors may include video inputs to receive image data from multiple image sensors, and they may also include video output functions. In one example, the EyeQ2 uses ® the 90 nm micron technology, which operates at 332 MHz. The EyeQ2 ® architecture consists of two 32-bit RISC CPUs with floating point and hyperthreading (MIPS32 ® 34K ® cores), five Vision Computing Engines (VCE), three Vector Microcode Processors (VMP ®), Denali 64-bit Mobile DDR controller, 128-bit internal Sonics interconnect, dual 16-bit video input and 18-bit video output controllers, 16 channels of DMA, and various peripherals. The MIPS34K CPU manages the five VCEs, three VMP™ and the DMA, the second MIPS34K CPU and the multi-channel DMA, as well as the other peripherals. The five VCEs, three VMP ® and the MIPS34K CPU can perform intensive image processing calculations required for applications with multifunction bundles. In another example, the EyeQ3 ® , a third-generation processor that is six times more powerful than the EyeQ2 ® , may be used in the disclosed embodiments. In other examples, the EyeQ4 ® and / or the EyeQ5 ®be used in the disclosed embodiments. Of course, any newer or future EyeQ processing devices may also be used in conjunction with the disclosed embodiments.

[0022] Each of the processing devices disclosed herein may be configured to perform particular processes. Configuring a processing device, such as one of the described EyeQ processors or another controller or microprocessor, to perform particular functions may include programming computer-executable instructions and providing those instructions to the processing device for execution during operation of the processing device. In some embodiments, configuring a processing device may include directly programming the processing device with architectural instructions. In other embodiments, configuring a processing device may include storing executable instructions in memory that the processing device can access during operation.For example, the processing device may access the memory to obtain and execute the stored instructions during operation. In any event, the processing device configured to perform the sensing, image analysis, and / or navigation functions disclosed herein is a specialized hardware-based system for controlling multiple hardware-based components of a host vehicle.

[0023] Fig. 1 illustrates two separate processing devices included within processing unit 110, but more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to perform the tasks of application processor 180 and image processing processor 190. In other embodiments, these tasks may be performed by more than two processing devices. Furthermore, in some embodiments, system 100 may include one or more processing units 110 without including other components, such as image acquisition component 120.

[0024] Processing unit 110 may include various types of devices. For example, processing unit 110 may include various devices such as a controller, an image preprocessor, a central processing unit (CPU), support circuitry, digital signal processors, integrated circuits, memory, or other types of devices for image processing and analysis. The image preprocessing processor may include a video processor that acquires, digitizes, and processes the images from the image sensors. The CPU may include any number of microcontrollers or microprocessors. The support circuitry may be any number of circuits well known in the art, including cache, power, clock, and input / output circuits. Memory may store software that, when executed by the processor, controls the operation of the system.The storage may include databases and image processing software. The storage may include any number of random access memories, read-only memories, flash memories, disk drives, optical storage, tape storage, removable storage, and other types of storage. In one case, the storage may be separate from the processing unit 110. In another case, the storage may be integrated into the processing unit 110.

[0025] Each memory 140, 150 may include software instructions that, when executed by a processor (e.g., the application processor 180 and / or the image processor 190), may control the operation of various aspects of the system 100. These storage units may include various databases and image processing software, as well as a trained system, such as a neural network or a deep neural network. The storage units may include random access memory, read-only memory, flash memory, disk drives, optical storage, tape storage, removable storage, and / or other types of storage. In some embodiments, the storage units 140, 150 may be separate from the application processor 180 and / or the image processor 190. In other embodiments, these storage units may be integrated into the application processor 180 and / or the image processor 190.

[0026] Position sensor 130 may comprise any type of device capable of determining a position associated with at least one component of system 100. In some embodiments, position sensor 130 may comprise a GPS receiver. Such receivers may determine user position and velocity by processing signals transmitted by the Global Positioning System satellites. Position information from position sensor 130 may be provided to application processor 180 and / or image processor 190.

[0027] In some embodiments, system 100 may include components such as a speed sensor (e.g., a speedometer) for measuring the speed of vehicle 200. System 100 may also include one or more acceleration sensors (either single-axis or multi-axis) for measuring the accelerations of vehicle 200 along one or more axes.

[0028] The storage units 140, 150 may include a database or otherwise organized data indicating a location of known landmarks. Sensory information (such as images, radar signals, depth information from lidar, or stereo processing of two or more images) of the environment may be processed along with position information, such as a GPS coordinate, the vehicle's ego motion, etc., to determine the vehicle's current position relative to the known landmarks and to refine the vehicle position. Certain aspects of this technology are incorporated into a localization technology called REM™, which is marketed by the assignee of the present application.

[0029] The user interface 170 may include any device capable of providing information to or receiving input from one or more users of the system 100. In some embodiments, the user interface 170 may include user input devices, such as a touchscreen, a microphone, a keyboard, pointing devices, track wheels, cameras, knobs, buttons, etc. With such input devices, a user may provide input or commands to the system 100 by typing instructions or information, issuing voice commands, selecting menu options on a screen using buttons, pointers, or tracking systems, or employing other suitable techniques for communicating information with the system 100.

[0030] The user interface 170 may be equipped with one or more processing devices configured to provide and receive information to or from a user and to process that information for use by, for example, the application processor 180. In some embodiments, such processing devices may execute instructions to detect and track eye movements, receive and interpret voice commands, detect and interpret touches and / or gestures on a touchscreen, respond to keyboard inputs or menu selections, etc. In some embodiments, the user interface 170 may include a display, a speaker, a tactile device, and / or other devices for providing information to the user.

[0031] The map database 160 may include any type of database for storing map data useful to the system 100. In some embodiments, the map database 160 may include data relating to the position of various items in a reference coordinate system, for example, roads, bodies of water, geographical features, businesses, points of interest, restaurants, gas stations, etc. The map database 160 may store not only the locations of such items, but also descriptors relating to those items, including, for example, the names relating to one of the stored features. In some embodiments, the map database 160 may be physically located with other components of the system 100. Alternatively or additionally, the map database 160, or a portion thereof, may be located at a remote location relative to other components of the system 100 (e.g., processing unit 110).In such embodiments, information may be downloaded from the map database 160 via a wired or wireless data connection to a network (e.g., via a cellular network and / or the Internet, etc.). In some cases, the map database 160 may store a sparse data model that includes polynomial representations of certain road features (e.g., lane markers) or target trajectories for the host vehicle. The map database 160 may also include stored representations of various detected landmarks that can be used to determine or update a known position of the host vehicle relative to a target trajectory. The representations of the landmarks may include data fields such as the type of landmark, the location of the landmark, and other possible identifiers.

[0032] The image capture devices 122, 124, and 126 may include any type of device suitable for capturing at least one image from an environment. Furthermore, any number of image capture devices may be used to capture images for input to the image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, or even four or more image capture devices. The image capture devices 122, 124, and 126 are further described with reference to Fig. 2B-2E described below.

[0033] One or more cameras (e.g., image capture devices 122, 124, and 126) may be part of a sensing block included in a vehicle. Various other sensors may be included in the sensor block, and any or all of the sensors may be used to detect the vehicle's navigation status. In addition to the cameras (forward, side, rear, etc.), other sensors such as radar, lidar, and acoustic sensors may also be included in the sensor block. Additionally, the sensor block may include one or more components configured to communicate and transmit / receive information related to the vehicle's surroundings. Such components may include, for example, wireless transceivers (RF, etc.) that can receive sensor-based information from a source remote from the host vehicle, or any other type of information about the host vehicle's surroundings.This information may include information about sensor output or related information received from vehicle systems other than the host vehicle. In some embodiments, such information may include information received from a remote computing device, a central server, etc. Furthermore, the cameras may take many different configurations: single units, multiple cameras, camera clusters, long FOV, short FOV, wide angle, fisheye, etc.

[0034] The system 100 or various components thereof may be integrated into various platforms. In some embodiments, the system 100 may be incorporated into a vehicle 200, as shown in Fig. 2A. For example, the vehicle 200 may be equipped with a processing unit 110 and one of the other components of the system 100 as described above in Fig. 1. While in some embodiments, the vehicle 200 may be equipped with only a single image capturing device (e.g., a camera), in other embodiments, as described in connection with Fig. 2B-2E, multiple image capture devices may be used. For example, one of the two image capture devices 122 and 124 of the vehicle 200, as shown in Fig. 2A, may be part of an advanced driver assistance system (ADAS) kit.

[0035] The image capture devices included in the vehicle 200 as part of the image capture unit 120 can be positioned at any suitable location. In some embodiments, as shown in Fig. As shown in Figures 2A-2E and 3A-3C, the image capture device 122 may be located near the rearview mirror. This position may provide a line of sight similar to that of the driver of the vehicle 200, allowing for determining what is and is not visible to the driver. The image capture device 122 may be positioned anywhere near the rearview mirror, but placing the image capture device 122 on the driver's side of the mirror may further assist in obtaining images representative of the driver's field of view and / or line of sight.

[0036] Other locations for the image capture devices of image acquisition unit 120 may also be used. Image capture device 124 may be located, for example, on or in the bumper of vehicle 200. Such a location is particularly suitable for image capture devices that have a large field of view. The line of sight of bumper-mounted image capture devices may differ from that of the driver, so the bumper-mounted image capture device and the driver may not always see the same objects. The image capture devices (e.g., image capture devices 122, 124, and 126) may also be located in other locations.The image capture devices may be located, for example, on or in one or both side mirrors of the vehicle 200, on the roof of the vehicle 200, on the hood of the vehicle 200, on the trunk of the vehicle 200, on the sides of the vehicle 200, on, behind, or in front of one of the windows of the vehicle 200, and in or near lighting fixtures at the front and / or rear of the vehicle 200, etc.

[0037] In addition to the image capture devices, the vehicle 200 may include various other components of the system 100. For example, the processing unit 110 may be either integrated into the vehicle 200 or enclosed separately from the vehicle's control unit (ECU). The vehicle 200 may also be equipped with a position sensor 130, such as a GPS receiver, and may also include a map database 160 and storage units 140 and 150.

[0038] As previously mentioned, the wireless transceiver 172 may receive data via one or more networks (e.g., cellular networks, the Internet, etc.). For example, the wireless transceiver 172 may upload the data collected by the system 100 to one or more servers and download data from one or more servers. For example, the wireless transceiver 172 may allow the system 100 to receive periodic or on-demand updates to the data stored in the map database 160, the memory 140, and / or the storage 150. Similarly, the wireless transmit and receive unit 172 may upload all data (e.g., images captured by the image acquisition unit 120, data received from the position sensor 130 or other sensors, vehicle control systems, etc.) from the system 100 and / or all data processed by the control unit 110 to one or more servers.

[0039] System 100 may upload data to a server (e.g., to the cloud) based on a set level of privacy. For example, system 100 may implement privacy settings to regulate or restrict the types of data (including level metadata) sent to the server that can uniquely identify a vehicle or the driver / owner of a vehicle. Such settings may be set by the user, for example, via wireless transceiver 172, initialized by factory default settings, or performed by data received by wireless transceiver 172.

[0040] In some embodiments, the system 100 may upload data according to a "high" privacy level, and when configured, the system 100 may transmit data (e.g., location information related to a route, captured images, etc.) without any details about the specific vehicle and / or driver / owner. For example, when uploading data according to a "high" privacy setting, the system 100 may not include a vehicle identification number (VIN) or a name of a driver or owner of the vehicle, and instead transmit information such as captured images and / or limited location information related to a route.

[0041] Other levels of privacy are also contemplated. For example, system 100 may transmit data to a server at a "medium" level and include additional information not included in the "high" level, such as the make and / or model of a vehicle and / or the type of vehicle (e.g., a passenger car, an SUV, a truck, etc.). In some embodiments, system 100 may upload data at a "low" privacy level. At a "low" privacy level, system 100 may upload data and include information sufficient to uniquely identify a particular vehicle, the owner / driver, and / or a portion or the entire route traveled by the vehicle.Such “low” level data may include, for example, the vehicle identification number, the name of the driver / owner, the place of origin of the vehicle before departure, the intended destination of the vehicle, the make and / or model of the vehicle, the type of vehicle, etc.

[0042] Fig. 2A is a schematic side view of an exemplary vehicle imaging system in accordance with the disclosed embodiments. Fig. Figure 2B is a schematic illustration of the Fig. 2A shown embodiment in plan view. As shown in Fig. 2B, the disclosed embodiments may include a vehicle 200 including in its body a system 100 having a first image capture device 122 positioned near the rearview mirror and / or near the driver of the vehicle 200, a second image capture device 124 positioned on or in a device region (e.g., one of the device regions 210) of the vehicle 200, and a processing unit 110.

[0043] As in Fig. 2C, the image capture devices 122 and 124 may both be positioned near the rearview mirror and / or near the driver of the vehicle 200. Although in Fig. 2B and Fig. 2C, two image capture devices 122 and 124 are shown, it is to be understood that other embodiments may include more than two image capture devices. In the Fig. 2D and Fig. For example, in the embodiments shown in Figure 2E, first, second, and third image capture devices 122, 124, and 126 are included in the system 100 of the vehicle 200.

[0044] As in Fig. 2D, the image capture device 122 may be positioned near the rearview mirror and / or near the driver of the vehicle 200, and the image capture devices 124 and 126 may be positioned on or in a region of the bumper (e.g., one of the bumper regions 210) of the vehicle 200. And as in Fig. As shown in Figure 2E, the image capture devices 122, 124, and 126 may be positioned near the rearview mirror and / or near the driver's seat of the vehicle 200. The disclosed embodiments are not limited to a particular number and configuration of the image capture devices, and the image capture devices may be positioned at any suitable location in and / or on the vehicle 200.

[0045] It should be understood that the disclosed embodiments are not limited to vehicles and may be applied in other contexts. It should also be understood that the disclosed embodiments are not limited to any particular type of vehicle 200 and may be applicable to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.

[0046] The first image capture device 122 may include any suitable type of image capture device. The image capture device 122 may include an optical axis. In one case, the image capture device 122 may include an Aptina M9V024 WVGA sensor with a global shutter. In other embodiments, the image capture device 122 may provide a resolution of 1280 × 960 pixels and include a rolling shutter. The image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included, for example, to provide a desired focal length and field of view for the image capture device. In some embodiments, the image capture device 122 may be associated with a 6 mm or a 12 mm lens.In some embodiments, the image capture device 122 may be configured to capture images having a desired field of view (FOV) 202, as shown in FIG. Fig. 2D. For example, the image capture device 122 may be configured to have a regular FOV, such as in a range of 40 degrees to 56 degrees, including a 46-degree FOV, a 50-degree FOV, a 52-degree FOV, or more. Alternatively, the image capture device 122 may be configured to have a narrow FOV in the range of 23 to 40 degrees, such as a 28-degree or 36-degree FOV. Furthermore, the image capture device 122 may have a wide FOV in the range of 100 to 180 degrees. In some embodiments, the image capture device 122 may include a wide-angle bumper camera or a camera with a field of view of up to 180 degrees. In some embodiments, the image capture device 122 may be a 7.2 megapixel image capture device with an aspect ratio of about 2:1 (e.g., H×V=3800×1900 pixels) and a horizontal angle of view of about 100 degrees.Such an image pickup device can be used instead of a three-image pickup device configuration. Due to significant lens distortions, the vertical FOV of such an image pickup device can be significantly less than 50 degrees in implementations where the image pickup device uses a radially symmetric lens. For example, such a lens may not be radially symmetric, which would allow a vertical FOV of more than 50 degrees with a horizontal FOV of 100 degrees.

[0047] The first image capture device 122 may capture a plurality of first images relating to a scene associated with the vehicle 200. Each of the plurality of first images may be captured as a series of image lines that may be used with a rolling shutter. Each scan line may include a plurality of pixels.

[0048] The first image capture device 122 may have a scan rate associated with the acquisition of each of the first series of image lines. The scan rate may refer to a rate at which an image sensor can acquire image data associated with each pixel included in a particular scan line.

[0049] The image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, for example, CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be used in conjunction with a rolling shutter, so that each pixel in a row is read individually, and the rows are scanned line by line until an entire frame is captured. In some embodiments, the rows may be captured sequentially from top to bottom relative to the frame.

[0050] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124, and 126) may form a high-resolution imager and may have a resolution of greater than 5M pixels, 7M pixels, 10M pixels, or more.

[0051] The use of a rolling shutter can result in pixels in different rows being exposed and captured at different times, which can introduce distortion and other image artifacts in the captured frame. On the other hand, if the image capture device 122 is configured to operate with a global or synchronous shutter, all pixels can be exposed for the same time and during a common exposure period. Therefore, the image data in a frame collected by a system with a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a specific time. In contrast, in a rolling shutter application, each row of a frame is exposed, and the data is captured at different times. Thus, moving objects can appear distorted in an image capture device with a rolling shutter. This phenomenon is described in more detail below.

[0052] The second image capture device 124 and the third image capture device 126 may be any type of image capture device. Like the first image capture device 122, each of the image capture devices 124 and 126 may include an optical axis. In one embodiment, each of the image capture devices 124 and 126 may include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of the image capture devices 124 and 126 may include a rolling shutter. Like the image capture device 122, the image capture devices 124 and 126 may be configured to include various lenses and optical elements. In some embodiments, lenses associated with image capture devices 124 and 126 may provide FOVs (e.g., FOVs 204 and 206) that are equal to or narrower than an FOV (e.g., FOV 202) associated with image capture device 122.For example, the image capture devices 124 and 126 may have FOVs of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less.

[0053] Image capture devices 124 and 126 may capture a plurality of second and third images related to a scene associated with vehicle 200. Each of the plurality of second and third images may be captured as second and third series of image lines, which may be captured with a rolling shutter. Each line or row may include a plurality of pixels. Image capture devices 124 and 126 may have a second and third scan rate associated with the capture of each of the image lines included in the second and third series.

[0054] Each image capture device 122, 124, and 126 may be positioned at any suitable position and orientation relative to the vehicle 200. The relative positioning of the image capture devices 122, 124, and 126 may be selected to combine the information captured by the image capture devices. For example, in some embodiments, an FOV (such as FOV 204) associated with the image capture device 124 may partially or completely overlap with an FOV (such as FOV 202) associated with the image capture device 122 and an FOV (such as FOV 206) associated with the image capture device 126.

[0055] The image pickup devices 122, 124, and 126 may be located at any location on the vehicle 200 at a suitable height. In one case, there may be a height difference between the image pickup devices 122, 124, and 126 that provides sufficient parallax information for stereo analysis. For example, as shown in Fig. 2A, the two image pickup devices 122 and 124 are at different heights. There may also be a difference in lateral displacement between the image pickup devices 122, 124, and 126, which provides additional parallax information for stereo analysis by the processing unit 110, for example. The difference in lateral displacement can be calculated with the x be referred to as in Fig. 2C and Fig. 2D. In some embodiments, there may be a forward or backward offset (e.g., a range shift) between the image capture devices 122, 124, and 126. For example, the image capture device 122 may be 0.5 to 2 meters or more behind the image capture device 124 and / or the image capture device 126. This type of offset may allow one of the image capture devices to cover potential blind spots of the other image capture device(s).

[0056] The image capture devices 122 may have any resolution (e.g., the number of pixels associated with the image sensor), and the resolution of the image sensor(s) associated with the image capture device 122 may be higher, lower, or equal to the resolution of the image sensor(s) associated with the image capture devices 124 and 126. In some embodiments, the image sensor(s) associated with the image capture device 122 and / or the image capture devices 124 and 126 may have a resolution of 640×480, 1024×768, 1280×960, or another suitable resolution.

[0057] The frame rate (e.g., the rate at which an image capture device acquires a set of pixel data of one image frame before moving on to acquiring pixel data associated with the next image frame) may be controllable. The frame rate associated with image capture device 122 may be higher, lower, or the same as the frame rate associated with image capture devices 124 and 126. The frame rate associated with image capture devices 122, 124, and 126 may depend on a variety of factors that may affect the time of the frame rate. For example, one or more of image capture devices 122, 124, and 126 may include a selectable pixel delay time introduced before or after the acquisition of image data associated with one or more pixels of an image sensor in image capture device 122, 124, and / or 126.In general, the image data corresponding to each pixel may be acquired at a particular device clock rate (e.g., one pixel per clock cycle). In embodiments including a rolling shutter, one or more of the image capture devices 122, 124, and 126 may further include a selectable horizontal blanking period introduced before or after the acquisition of image data associated with a pixel row of an image sensor in the image capture device 122, 124, and / or 126. Furthermore, one or more of the image capture devices 122, 124, and / or 126 may include a selectable vertical blanking period introduced before or after the acquisition of image data associated with a frame of the image capture device 122, 124, and / or 126.

[0058] These timing controls can enable synchronization of the frame rates associated with image capture devices 122, 124, and 126, even if the scan rates of each device differ. As explained in more detail below, these selectable timing controls, among other factors (e.g., image sensor resolution, maximum line scan rates, etc.), can enable synchronization of image capture from an area where the FOV of image capture device 122 overlaps with one or more FOVs of image capture devices 124 and 126, even if the field of view of image capture device 122 differs from the FOVs of image capture devices 124 and 126.

[0059] The frame rate timing in image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensors. For example, if both devices have similar scan rates, and one device includes an image sensor with a resolution of 640 × 480 and another device includes an image sensor with a resolution of 1280 × 960, more time will be required to capture a frame of image data from the higher-resolution sensor.

[0060] Another factor that can affect the acquisition time of image data in image capture devices 122, 124, and 126 is the maximum line scan rate. For example, acquiring a series of image data from an image sensor included in image capture devices 122, 124, and 126 requires a certain minimum amount of time. Assuming no pixel delay times are added, this minimum amount of time for acquiring a series of image data is related to the maximum scan rate for a particular device. Devices that offer higher maximum line scan rates have the potential to provide higher frame rates than devices with lower maximum line scan rates. In some embodiments, one or more of image capture devices 124 and 126 may have a maximum line scan rate that is higher than the maximum line scan rate associated with image capture device 122.In some embodiments, the maximum line scan rate of device 124 and / or 126 may be 1.25, 1.5, 1.75, or 2 times or more the maximum line scan rate of image capture device 122.

[0061] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may operate at a scan rate less than or equal to its maximum scan rate. The system may be configured so that one or more of image capture devices 124 and 126 operate at a scan rate equal to the scan rate of image capture device 122. In other cases, the system may be configured so that the line scan rate of image capture device 124 and / or image capture device 126 is 1.25, 1.5, 1.75, or 2 times or more the line scan rate of image capture device 122.

[0062] In some embodiments, the image capture devices 122, 124, and 126 may be asymmetric. That is, they may include cameras with different fields of view (FOV) and focal lengths. The fields of view of the image capture devices 122, 124, and 126 may, for example, cover any desired area relative to the surroundings of the vehicle 200. In some embodiments, one or more of the image capture devices 122, 124, and 126 may be configured to capture image data from an environment in front of the vehicle 200, behind the vehicle 200, on the sides of the vehicle 200, or combinations thereof.

[0063] Furthermore, the focal length associated with each image capture device 122, 124, and / or 126 can be selected (e.g., by installing suitable lenses, etc.) so that each device captures images of objects at a desired distance relative to the vehicle 200. In some embodiments, the image capture devices 122, 124, and 126 can, for example, capture images of objects located only a few meters from the vehicle. The image capture devices 122, 124, and 126 can also be configured to capture images of objects farther from the vehicle (e.g., 25 m, 50 m, 100 m, 150 m, or more). Furthermore, the focal lengths of the image capture devices 122, 124, and 126 can be selected so that an image capture device (e.g., the image capture device 122) can capture images of objects that are relatively close to the vehicle (e.g.,within 10 m or within 20 m), while the other image pickup devices (e.g., image pickup devices 124 and 126) can capture images of objects that are farther away from the vehicle 200 (e.g., more than 20 m, 50 m, 100 m, 150 m, etc.).

[0064] According to some embodiments, the FOV of one or more of the image capture devices 122, 124, and 126 may have a wide angle. For example, it may be advantageous to have an FOV of 140 degrees, particularly for the image capture devices 122, 124, and 126 that may be used to capture images of the area near the vehicle 200. For example, the image capture device 122 may be used to capture images of the area to the right or left of the vehicle 200. In such cases, it may be desirable for the image capture device 122 to have a large FOV (e.g., at least 140 degrees).

[0065] The field of view associated with each image capture device 122, 124, and 126 may depend on the respective focal lengths. For example, as the focal length increases, the corresponding field of view decreases.

[0066] The image capture devices 122, 124, and 126 can be configured to have any suitable fields of view. In one particular example, the image capture device 122 may have a horizontal FOV of 46 degrees, the image capture device 124 may have a horizontal FOV of 23 degrees, and the image capture device 126 may have a horizontal FOV between 23 and 46 degrees. In another case, the image capture device 122 may have a horizontal FOV of 52 degrees, the image capture device 124 may have a horizontal FOV of 26 degrees, and the image capture device 126 may have a horizontal FOV between 26 and 52 degrees. In some embodiments, the ratio between the FOV of the image capture device 122 and the FOVs of the image capture device 124 and / or the image capture device 126 may be between 1.5 and 2.0. In other embodiments, this ratio may vary between 1.25 and 2.25.

[0067] The system 100 can be configured such that a field of view of the image capture device 122 at least partially or completely overlaps with a field of view of the image capture device 124 and / or the image capture device 126. In some embodiments, the system 100 can be configured such that the fields of view of the image capture devices 124 and 126, for example, are within (e.g., narrower than) the field of view of the image capture device 122 and have a common center with it. In other embodiments, the image capture devices 122, 124, and 126 can capture adjacent FOVs or have a partial overlap in their FOVs.In some embodiments, the fields of view of the image capture devices 122, 124, and 126 may be aligned such that the center of the image capture devices 124 and / or 126 with the narrower FOV is in the lower half of the field of view of the device 122 with the wider FOV.

[0068] Fig. Figure 2F is a schematic representation of exemplary vehicle control systems consistent with the disclosed embodiments. As shown in Fig. As indicated in Figure 2F, the vehicle 200 may include the throttle system 220, the braking system 230, and the steering system 240. The system 100 may provide inputs (e.g., control signals) to the throttle system 220, the braking system 230, and the steering system 240 via one or more data connections (e.g., one or more wired and / or wireless connections for transmitting data). Based on the analysis of images captured by the image capture devices 122, 124, and / or 126, the system 100 may, for example, provide control signals to the throttle system 220, the braking system 230, or the steering system 240 to navigate the vehicle 200 (e.g., by initiating acceleration, a turn, a lane shift, etc.). Furthermore, the system 100 may receive inputs from the throttle system 220, the braking system 230, or the steering system 24 that indicate the operating state of the vehicle 200 (e.g.,the speed, whether the vehicle is braking and / or turning, etc.). Further details will be provided in conjunction with . Fig. 4-7 provided below.

[0069] As in Fig. 3A, the vehicle 200 may also include a user interface 170 for interaction with a driver or a passenger of the vehicle 200. For example, in a vehicle application, the user interface 170 may include a touchscreen 320, knobs 330, buttons 340, and a microphone 350. A driver or passenger of the vehicle 200 may also use handles (e.g., on or near the steering column of the vehicle 200, including, for example, turn signal handles), buttons (e.g., on the steering wheel of the vehicle 200), and the like to interact with the system 100. In some embodiments, the microphone 350 may be positioned near a rearview mirror 310. In some embodiments, the image capture device 122 may also be located near the rearview mirror 310. In some embodiments, the user interface 170 may also include one or more speakers 360 (e.g., the speakers of a vehicle audio system).For example, the system 100 may provide various notifications (e.g., alarms) through the speakers 360.

[0070] Fig. 3B-3D are illustrations of an exemplary camera mount 370 configured to be positioned behind a rearview mirror (e.g., rearview mirror 310) and on the windshield of a vehicle, according to the disclosed embodiments. As shown in Fig. 3B, the camera mount 370 may include the image capture devices 122, 124, and 126. The image capture devices 124 and 126 may be positioned behind an anti-glare shield 380 that is flush with the vehicle windshield and includes a composition of film and / or anti-reflective materials. For example, the anti-glare shield 380 may be positioned to align with a vehicle windshield having a corresponding slope. In some embodiments, each of the image capture devices 122, 124, and 126 may be positioned behind the anti-glare shield 380, such as in Fig. 3D. The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124, and 126, camera mount 370, and glare shield 380. Fig. Figure 3C illustrates the Fig. 3B shows an illustration of the camera mount 370 from a frontal perspective.

[0071] As one of ordinary skill in the art, having the benefit of this disclosure, will appreciate, numerous variations and / or modifications may be made to the above-disclosed embodiments. For example, not all components are essential to the operation of system 100. Further, each component may be disposed in any suitable location of system 100, and the components may be rearranged in a variety of configurations while providing the functionality of the disclosed embodiments. The above configurations are therefore only examples, and independent of the configurations described above, system 100 may provide a variety of functions to analyze the environment of vehicle 200 and control vehicle 200 in response to the analysis.

[0072] As discussed in detail below, and in accordance with various disclosed embodiments, system 100 may provide a variety of features related to autonomous driving and / or driver assistance technologies. For example, system 100 may analyze image data, position data (e.g., GPS location information), map data, speed data, and / or data from sensors included in vehicle 200. System 100 may collect the data for analysis from, for example, image capture unit 120, position sensor 130, and other sensors. Further, system 100 may analyze the collected data to determine whether or not vehicle 200 should perform a particular action and then automatically perform the particular action without human intervention.For example, when vehicle 200 is navigating without human intervention, system 100 may automatically control the braking, acceleration, and / or steering of vehicle 200 (e.g., by sending control signals to one or more of throttle systems 220, braking systems 230, and steering systems 240). Further, system 100 may analyze the collected data and issue warnings and / or alarms to the vehicle's occupants based on the analysis of the collected data. Further details regarding the various embodiments provided by system 100 are provided below. FORWARD-FACING MULTI-IMAGE SYSTEM

[0073] As described above, system 100 may provide driving assistance using a multi-camera system. The multi-camera system may use one or more cameras directed in the forward direction of a vehicle. In other embodiments, the multi-camera system may include one or more cameras directed toward the side of a vehicle or toward the rear of the vehicle. For example, in one embodiment, system 100 may use a two-camera imaging system where a first camera and a second camera (e.g., image capture device 122 and 124) may be positioned on the front and / or sides of a vehicle (e.g., vehicle 200). Other camera configurations are consistent with the disclosed embodiments, and the configurations disclosed herein are examples. For example, system 100 may configure any number of cameras (e.g., one, two, three, four, five, six, seven, eight, etc.).) Furthermore, the system 100 may include "clusters" of cameras. For example, a group of cameras (including any number of cameras, e.g., one, four, eight, etc.) may be facing forward relative to the vehicle or in any other direction (e.g., toward the reward, to the side, at an angle, etc.). Accordingly, the system 100 may include multiple groups of cameras, with each group facing a particular direction to capture images from a particular region of the vehicle's surroundings.

[0074] The first camera may have a field of view that is larger or smaller than, or partially overlaps, the field of view of the second camera. Furthermore, the first camera may be connected to a first image processor to perform monocular image analysis on the images provided by the first camera, and the second camera may be connected to a second image processor to perform monocular image analysis on the images provided by the second camera. The outputs (e.g., processed information) of the first and second image processors may be combined. In some embodiments, the second image processor may receive images from both the first camera and the second camera to perform stereo analysis. In another embodiment, system 100 may use a three-camera imaging system where each of the cameras has a different field of view.Such a system can therefore make decisions based on information from objects located at different distances both in front of the vehicle and to the sides of the vehicle. The term "monocular image analysis" can refer to cases where image analysis is performed based on images acquired from a single viewpoint (e.g., from a single camera). Stereo image analysis can refer to cases where image analysis is performed based on two or more images acquired with one or more variations of an image acquisition parameter. Images suitable for performing stereo image analysis can, for example, include images acquired from two or more different positions, from different viewpoints, with different focal lengths, along with parallax information, etc.

[0075] For example, in one embodiment, system 100 may implement a three-camera configuration using image capture devices 122-126. In such a configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees or other values ​​from a range of about 20 to 45 degrees, etc.), image capture device 124 may provide a wide field of view (e.g., 150 degrees or other values ​​from a range of about 100 to about 180 degrees), and image capture device 126 may provide a medium field of view (e.g., 46 degrees or other values ​​from a range of about 35 to about 60 degrees). In some embodiments, image capture device 126 may function as the main or primary camera. Image capture devices 122-126 may be positioned in front of rearview mirror 310 and substantially adjacent to each other (e.g., 6 cm apart).Furthermore, in some embodiments, as described above, one or more of the image capture devices 122-126 may be mounted behind the glare shield 380, which is flush with the windshield of the vehicle 200. Such a shield may help minimize the effects of reflections from the vehicle interior on the image capture devices 122-126.

[0076] In another embodiment, as described above in connection with Fig. 3B and Fig. As explained in Figure 3C, the wide field of view camera (e.g., image capture device 124 in the example above) may be mounted lower than the narrow and wide field of view cameras (e.g., image capture devices 122 and 126 in the example above). This configuration may provide a clear line of sight from the wide field of view camera. To reduce reflections, the cameras may be mounted near the windshield of the vehicle 200 and include polarizers on the cameras to attenuate the reflected light.

[0077] A three-camera system may provide certain performance characteristics. For example, some embodiments may include the ability to validate the detection of objects by one camera based on the detection results of another camera. In the three-camera configuration described above, the processor unit 110 may, for example, include three processing units (e.g., three EyeQ series processor chips as described above), with each processing unit dedicated to processing images captured by one or more image capture devices 122-126.

[0078] In a three-camera system, a first processing device may receive images from both the main camera and the narrow FOV camera and perform image processing of the narrow FOV camera to detect, for example, other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other objects on the road. Furthermore, the first processing device may calculate a pixel disparity between the images from the main camera and the narrow FOV camera and create a 3D reconstruction of the surroundings of the vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data or with 3D information calculated based on information from another camera.

[0079] The second processing device can receive images from the main camera and perform a process for detecting other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other objects on the road. Additionally, the second processing device can calculate a camera displacement and, based on the displacement, calculate a pixel disparity between consecutive images and create a 3D reconstruction of the scene (e.g., a structure from motion). The second processing device can send the structure from the motion-based 3D reconstruction to the first processing device to combine it with the 3D stereo images.

[0080] The third processing device may receive images from the wide-angle camera (FOV) and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other objects on the road. The third processing device may further execute additional instructions for analyzing images to identify moving objects in the image, such as vehicles changing lanes, pedestrians, etc.

[0081] In some embodiments, streams of image-based information that are independently captured and processed may include a means of providing redundancy within the system. Such redundancy may, for example, include using a first image capture device and the images processed by that device to validate and / or supplement information obtained by capturing and processing image information from at least one second image capture device.

[0082] In some embodiments, system 100 may utilize two image capture devices (e.g., image capture data 122 and 124) to assist in navigation of vehicle 200 and a third image capture device (e.g., image capture data 126) to provide redundancy and validate the analysis of the data received from the other two image capture devices. In such a configuration, for example, image capture devices 122 and 124 may provide images for stereo analysis by system 100 to assist in navigation of vehicle 200, while image capture device 126 may provide images for monocular analysis by system 100 to provide redundancy and validation of the information obtained based on the images captured by image capture device 122 and / or image capture device 124.That is, image capture device 126 (and a corresponding processing device) can be considered a redundant subsystem that provides verification of the analysis derived from image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AEB) system). Furthermore, in some embodiments, the redundancy and validation of the received data can be supplemented based on information received from one or more sensors (e.g., radar, lidar, acoustic sensors, information received from one or more transceivers external to a vehicle, etc.).

[0083] One skilled in the art will recognize that the above-mentioned camera configurations, camera placements, number of cameras, camera locations, etc., are merely examples. These and other components described with respect to the overall system may be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of a multi-camera system to provide driver assistance and / or autonomous vehicle functions are explained below.

[0084] Fig. Figure 4 is an exemplary functional block diagram of memory 140 and / or 150, which may be stored / programmed with instructions for performing one or more operations in accordance with the disclosed embodiments. Although the following refers to memory 140, one of ordinary skill in the art will recognize that instructions may also be stored in memory 140 and / or 150.

[0085] As in Fig. 4, memory 140 may store a monocular image analysis module 402, a stereo image analysis module 404, a velocity and acceleration measurement module 406, and a navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Further, application processor 180 and / or image processor 190 may execute the instructions stored in any of the modules 402-408 included in memory 140. One of ordinary skill in the art will understand that references to processing unit 110 in the following discussion may refer to applications, processing unit 180, and image processor 190 individually or collectively. Accordingly, the steps of any of the following processes may be performed by one or more processing devices.

[0086] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform a monocular image analysis of a set of images acquired by one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from a set of images with additional sensory information (e.g., information from radar) to complete the monocular image analysis. As described in connection with Fig. 5A-5D below, the monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exits, traffic signals, hazardous objects, and any other feature associated with a vehicle's environment. Based on the analysis, the system 100 (e.g., via the processing unit 110) may trigger one or more navigation responses in the vehicle 200, such as a turn, a lane shift, a change in acceleration, and the like, as described below in connection with the navigation response module 408.

[0087] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform a monocular image analysis of a set of images acquired by one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from a set of images with additional sensory information (e.g., information from radar, lidar, etc.) to complete the monocular image analysis. As described in connection with Fig. 5A-5D below, the monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exits, traffic signals, hazardous objects, and any other feature associated with a vehicle's environment. Based on the analysis, the system 100 (e.g., via the processing unit 110) may trigger one or more navigational responses in the vehicle 200, such as a turn, a lane change, a change in acceleration, and the like, as described further below in connection with determining a navigational response.

[0088] In one embodiment, the stereo image analysis module 404 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform a stereo image analysis of a first and a second set of images acquired by a combination of image capture devices selected from one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the first and second sets of images with additional sensory information (e.g., information from radar) to perform the stereo image analysis. For example, the stereo image analysis module 404 may include instructions for performing a stereo image analysis based on a first set of images acquired by the image capture device 124 and a second set of images acquired by the imaging device 126.As in connection with . Fig. 6 below, the stereo image analysis module 404 may include instructions for detecting a set of features in the first and second image sets, such as lane markings, vehicles, pedestrians, road markings, highway exits, traffic lights, hazardous objects, and the like. Based on the analysis, the processing unit 110 may trigger one or more navigational responses in the vehicle 200, such as a turn, a lane change, a change in acceleration, and the like, as described below in connection with the navigational response module 408. Furthermore, in some embodiments, the stereo image analysis module 404 may implement techniques associated with a trained system (such as a neural network or a deep neural network) or an untrained system.

[0089] In one embodiment, the speed and acceleration module 406 may store software configured to analyze data received from one or more computing and electromechanical devices within the vehicle 200 that are configured to cause a change in the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 may execute instructions associated with the speed and acceleration module 406 to calculate a target speed for the vehicle 200 based on data resulting from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404.Such data may include, for example, a target position, speed, and / or acceleration, the position and / or speed of the vehicle 200 relative to a nearby vehicle, pedestrian, or object on the road, position information for the vehicle 200 relative to lane markings on the road, and the like. Furthermore, the processing unit 110 may calculate a target speed for the vehicle 200 based on sensory inputs (e.g., information from radar) and inputs from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and / or the steering system 240 of the vehicle 200.Based on the calculated speed, the processing unit 110 may transmit electronic signals to the throttle system 220, the braking system 230, and / or the steering system 240 of the vehicle 200 to trigger a change in speed and / or acceleration, for example, by physically applying the brake or releasing the accelerator pedal of the vehicle 200.

[0090] In one embodiment, the navigation response module 408 may store software executable by the processing unit 110 to determine a desired navigation response based on data resulting from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include position and speed information associated with nearby vehicles, pedestrians, and objects, as well as information about the target position of the vehicle 200, and the like.Additionally, in some embodiments, the navigation response may be based (in part or in full) on map data, a predetermined position of the vehicle 200, and / or a relative velocity or acceleration between the vehicle 200 and one or more objects detected during execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 may also determine a desired navigation response based on sensory inputs (e.g., information from radar) and inputs from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and the steering system 240 of the vehicle 200.Based on the desired navigational response, processing unit 110 may transmit electronic signals to the throttle system 220, the braking system 230, and the steering system 240 of the vehicle 200 to trigger a desired navigational response, for example, by rotating the steering wheel of the vehicle 200 to achieve a rotation through a predetermined angle. In some embodiments, processing unit 110 may use the output of the navigation response module 408 (e.g., the desired navigational response) as input for executing the speed and acceleration module 406 to calculate a change in the speed of the vehicle 200.

[0091] Furthermore, each module disclosed herein (e.g., modules 402, 404, and 406) may implement techniques associated with a trained system (such as a neural network or a deep neural network) or an untrained system.

[0092] Fig. 5A is a flowchart illustrating an exemplary process 500A for triggering one or more navigational responses based on monocular image analysis in accordance with disclosed embodiments. In step 510, the processing unit 110 may receive a plurality of images via the data interface 128 between the processing unit 110 and the image acquisition unit 120. For example, a camera in the image acquisition unit 120 (e.g., the image acquisition device 122 with the field of view 202) may capture a plurality of images of an area in front of the vehicle 200 (or, e.g., on the sides or rear of a vehicle) and transmit them to the data processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). The processing unit 110 may execute the monocular image analysis module 402 to analyze the plurality of images in step 520, as described in connection with Fig. 5B-5D are described in detail below. By performing the analysis, processing unit 110 may identify a set of features within the set of images, such as lane markings, vehicles, pedestrians, road markings, highway exit ramps, traffic lights, and the like.

[0093] The processing unit 110 may also execute the monocular image analysis module 402 to detect various road hazards, such as pieces of a truck tire, fallen traffic signs, loose cargo, small animals, and the like, at step 520. Road obstacles may vary in texture, shape, size, and color, which may make such obstacles difficult to detect. In some embodiments, the processing unit 110 may execute the monocular image analysis module 402 to perform a multiple-frame analysis of the plurality of images to detect road hazards. For example, the processing unit 110 may estimate camera motion between consecutive frames and calculate the disparities in the pixels between frames to create a 3D map of the road.The processing unit 110 can then use the 3D map to identify the road surface as well as the hazards present above the road surface.

[0094] In step 530, the processing unit 110 may execute the navigation response module 408 to determine, based on the analysis performed in step 520 and the information provided above in connection with Fig. 4 to trigger one or more navigation responses in the vehicle 200. The navigation responses may include, for example, a turn, a lane change, a change in acceleration, and the like. In some embodiments, the processing unit 110 may use data resulting from the execution of the speed and acceleration module 406 to trigger one or more navigation responses. Furthermore, multiple navigation responses may occur simultaneously, in sequence, or in any combination. For example, the processing unit 110 may cause the vehicle 200 to change lanes and then accelerate, for example, by sequentially sending control signals to the steering system 240 and the throttle system 220 of the vehicle 200.Alternatively, the processing unit 110 may cause the vehicle 200 to brake and change lanes simultaneously, for example, by simultaneously sending control signals to the braking system 230 and the steering system 240 of the vehicle 200.

[0095] Fig. 5B is a flowchart illustrating an exemplary process 500B for detecting one or more vehicles and / or pedestrians in a set of images consistent with the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500B. In step 540, processing unit 110 may determine a set of candidate objects representing possible vehicles and / or pedestrians. For example, processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify possible locations in each image that may contain objects of interest (e.g., vehicles, pedestrians, or portions thereof). The predetermined patterns may be designed to achieve a high false hit rate and a low false miss rate.For example, processing unit 110 may use a low threshold for similarity to predetermined patterns to identify candidate objects as possible vehicles or pedestrians. This allows processing unit 110 to reduce the likelihood that an object representing a vehicle or pedestrian will be missed (e.g., because it was not identified).

[0096] In step 542, processing unit 110 may filter the set of candidate objects to exclude certain candidates (e.g., irrelevant or less relevant items) based on classification criteria. Such criteria may be derived from various properties associated with object types stored in a database (e.g., a database stored in memory 140). The properties may include the object's shape, dimensions, texture, position (e.g., relative to vehicle 200), and the like. Thus, processing unit 110 may use one or more sets of criteria to exclude false candidates from the set of candidate objects.

[0097] In step 544, processing unit 110 may analyze multiple frames of images to determine whether the objects in the candidate object set represent vehicles and / or pedestrians. For example, processing unit 110 may track a detected object across multiple consecutive frames and collect data associated with the object (e.g., size, position relative to vehicle 200, etc.) frame by frame. Additionally, processing unit 110 may estimate parameters for the detected object and compare the predicted data of the object's position in the frame to a predicted position.

[0098] In step 546, processing unit 110 may create a set of measurements for the detected objects. Such measurements may include, for example, position, velocity, and acceleration values ​​(relative to vehicle 200) associated with the detected objects. In some embodiments, processing unit 110 may construct the measurement data based on estimation methods that use a set of time-based observations, such as Kalman filters or linear quadratic estimation (LQE), and / or based on available modeling data for various object types (e.g., cars, trucks, pedestrians, bicycles, traffic signs, etc.). The Kalman filters may be based on a measurement of the scale of an object, where the measurement of the scale is proportional to a time to collision (e.g., the amount of time it takes vehicle 200 to reach the object).By performing steps 540-546, processing unit 110 may identify vehicles and pedestrians appearing in the set of captured images and derive information (e.g., position, speed, size) associated with the vehicles and pedestrians. Based on the identification and the derived information, processing unit 110 may trigger one or more navigation responses in vehicle 200, as described in connection with [ ]. Fig. 5A described above.

[0099] In step 548, the processing unit 110 may perform optical flow analysis of one or more images to reduce the likelihood of detecting a "false hit" and missing an object representing a vehicle or pedestrian. Optical flow analysis may, for example, refer to analyzing motion patterns relative to the vehicle 200 in one or more images associated with other vehicles and pedestrians that are distinct from the motion on the road surface. The processing unit 110 may calculate the motion of the candidate objects by observing the different positions of the objects across multiple image frames captured at different times. The processing unit 110 may use the position and time values ​​as inputs to mathematical models to calculate the motion of the objects.Thus, optical flow analysis may provide another method for detecting vehicles and pedestrians that are near vehicle 200. Processing unit 110 may perform optical flow analysis in combination with steps 540-546 to provide redundancy in detecting vehicles and pedestrians and increase the reliability of system 100.

[0100] Fig. 5C is a flowchart illustrating an example process 500C for detecting road markings and / or lane geometry information in a set of images, consistent with disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500C. In step 550, processing unit 110 may detect a set of objects by scanning one or more images. To detect segments of lane markings, lane geometry information, and other relevant road markings, processing unit 110 may filter the set of objects to exclude those deemed irrelevant (e.g., minor potholes, small rocks, etc.). In step 552, processing unit 110 may group together the segments identified in step 550 that belong to the same road marking or lane marking.Based on the grouping, the processing unit 110 may develop a model to represent the detected segments, such as a mathematical model.

[0101] In step 554, the processing unit 110 may create a set of measurements associated with the detected segments. In some embodiments, the processing unit 110 may create a projection of the detected segments from the image plane to the real plane. The projection may be characterized by a third-order polynomial having coefficients corresponding to physical properties such as position, slope, curvature, and curvature derivative of the detected road. In generating the projection, the processing unit 110 may consider changes in the road surface as well as the pitch and roll rates associated with the vehicle 200. Furthermore, the processing unit 110 may model the road elevation by analyzing the position and motion cues present on the road surface.Further, the processing unit 110 may estimate the pitch and roll rates associated with the vehicle 200 by adjusting a set of feature points in one or more images.

[0102] In step 556, the processing unit 110 may perform a multi-image analysis, for example, by tracking the detected segments across consecutive frames and collecting the data associated with the detected segments frame by frame. As the processing unit 110 performs a multi-frame analysis, the set of measurements created in step 554 may become more reliable and assigned to an increasingly higher level. Thus, in the forming steps 550-556, the processing unit 110 may identify road markings that appear in the set of captured images and derive lane geometry information. Based on the identification and the derived information, the processing unit 110 may trigger one or more navigation responses in the vehicle 200, as described in connection with Fig. 5A described above.

[0103] In step 558, the processing unit 110 may consider additional information sources to further develop a safety model for the vehicle 200 in the context of its environment. The processing unit 110 may use the safety model to define a context in which the system 100 can safely perform autonomous control of the vehicle 200. To develop the safety model, in some embodiments, the processing unit 110 may consider the position and movement of other vehicles, detected road edges and barriers, and / or general descriptions of the road shape extracted from map data (e.g., data from the map database 160). By considering additional information sources, the processing unit 110 may provide redundancy for detecting road markings and lane geometries and increase the reliability of the system 100.

[0104] Fig. 5D is a flowchart illustrating an exemplary process 500D for detecting traffic lights in a set of images consistent with the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500D. In step 560, processing unit 110 may scan the set of images and identify objects that appear at locations in the images likely to contain traffic lights. For example, processing unit 110 may filter the identified objects to establish a set of candidate objects, excluding those objects that are unlikely to correspond to a traffic light. The filtering may be based on various properties associated with a traffic light, such as shape, dimensions, texture, on-position (e.g., relative to vehicle 200), and the like.Such properties may be based on multiple examples of traffic lights and control signals and stored in a database. In some embodiments, processing unit 110 may perform a multi-frame analysis on the set of candidate objects that reflect possible traffic lights. For example, processing unit 110 may track the objects in consecutive frames, estimate the real-world position of the objects, and filter out the moving objects (which are unlikely to be traffic lights). In some embodiments, processing unit 110 may perform a color analysis of the candidate objects and identify the relative position of the detected colors that appear within possible traffic lights.

[0105] In step 562, processing unit 110 may analyze the geometry of a turn. The analysis may be based on any combination of: (i) the number of lanes detected on either side of vehicle 200, (ii) markers detected on the road (e.g., arrow markers), and (iii) descriptions of the turn extracted from map data (e.g., data from map database 160). Processing unit 110 may perform the analysis using information obtained from the execution of monocular analysis module 402. Furthermore, processing unit 110 may determine a correspondence between the traffic lights detected in step 560 and the lanes appearing near vehicle 200.

[0106] As the vehicle 200 approaches the junction, the processing unit 110 may, in step 564, update the confidence level associated with the analyzed geometry of the junction and the detected traffic lights. For example, the number of traffic lights expected at the junction compared to the actual number at the junction may influence the confidence level. Thus, based on the confidence level, the processing unit 110 may delegate control to the driver of the vehicle 200 to improve the safety state. When the processing unit 110 executes steps 560-564, it may identify the traffic lights in the set of captured images and analyze the information about the junction. Based on the identification and analysis, the processing unit 110 may trigger one or more navigation responses in the vehicle 200, as described in connection with Fig. 5A described above.

[0107] Fig. 5E is a flowchart illustrating an exemplary process 500E for triggering one or more navigation responses in vehicle 200 based on a vehicle path consistent with the disclosed embodiments. In step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. The vehicle path may be represented by a set of points expressed in coordinates (x, z), and the distance d ibetween two points in the set of points may range from 1 to 5 meters. In one embodiment, processing unit 110 may construct the initial vehicle path using two polynomials, such as a left and a right road polynomial. Processing unit 110 may calculate the geometric midpoint between the two polynomials and offset each point included in the resulting vehicle path by a predetermined offset (e.g., a smart lane offset), if present (a zero offset may correspond to traveling in the center of a lane). The offset may be in a direction perpendicular to a segment between any two points on the vehicle path.In another embodiment, the processing unit 110 may use a polynomial and an estimated lane width to offset each point of the vehicle path by half the estimated lane width plus a predetermined offset (e.g., a smart lane offset).

[0108] In step 572, the processing unit 110 may update the vehicle path created in step 570. The processing unit 110 may reconstruct the vehicle path constructed in step 570 with a higher resolution so that the distance d k between two points in the set of points representing the vehicle path is less than the distance d described above i . The distance d kFor example, it may be in the range of 0.1 to 0.3 meters. The processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm that provides a cumulative distance vector S corresponding to the total length of the vehicle path (i.e., based on the set of points representing the vehicle path).

[0109] In step 574, the processing unit 110 may determine a look-ahead point (expressed in coordinates as (x1, z1)) based on the updated vehicle path created in step 572. The processing unit 110 may extract the look-ahead point from the cumulative distance vector S, and the look-ahead point may be associated with a look-ahead distance and a look-ahead time. The look-ahead distance, which may have a lower limit of 10 to 20 meters, may be calculated as the product of the speed of the vehicle 200 and the look-ahead time. For example, as the speed of the vehicle 200 decreases, the look-ahead distance may also decrease (e.g., until it reaches the lower limit). The look-ahead time, which may be between 0.5 and 1.5 seconds, may be inversely proportional to the yield of one or more control circuits associated with a navigation response in the vehicle 200, such asThe course error tracking control loop. The yield of the course error tracking control loop can depend, for example, on the bandwidth of a yaw rate loop, a steering actuator loop, the vehicle's lateral dynamics, and so on. Therefore, the higher the yield of the course error tracking control loop, the shorter the look-ahead time.

[0110] In step 576, the processing unit 110 may determine a heading error and a yaw rate based on the look-ahead point determined in step 574. The processing unit 110 may determine the heading error by calculating the arctangent of the look-ahead point, e.g., arctan(x1 / z1). The processing unit 110 may determine the yaw rate command as the product of the heading error and a high level of gain. The gain of high level control may be equal to (2 / look-ahead time) if the look-ahead distance is not at the lower limit. Otherwise, the gain of high level control may be equal to (2 * vehicle speed 200 / look-ahead distance).

[0111] Fig. 5F is a flowchart illustrating an exemplary process 500F for determining whether a leading vehicle is changing lanes, in accordance with the disclosed embodiments. In step 580, the processing unit 110 may determine navigation information associated with a leading vehicle (e.g., a vehicle traveling in front of the vehicle 200). For example, the processing unit 110 may determine the position, speed (e.g., direction and speed), and / or acceleration of the leading vehicle using the techniques described in connection with Fig. 5A and Fig. 5B above. The processing unit 110 may also determine one or more road polynomials, a look-ahead point (associated with the vehicle 200), and / or a helical path (e.g., a set of points describing the path of the leading vehicle) using the techniques described in Fig. 5E described above.

[0112] In step 582, processing unit 110 may analyze the navigation information determined in step 580. In one embodiment, processing unit 110 may calculate the distance between a snail trail and a road polynomial (e.g., along the trail). If the deviation of this distance along the trail exceeds a predetermined threshold (e.g., 0.1 to 0.2 meters on a straight road, 0.3 to 0.4 meters on a moderately curved road, and 0.5 to 0.6 meters on a road with sharp turns), processing unit 110 may determine that the lead vehicle is likely changing lanes. If multiple vehicles are detected traveling in front of vehicle 200, processing unit 110 may compare the snail trails associated with each vehicle.Based on this comparison, processing unit 110 may determine that a vehicle whose snail trail does not match the snail trails of other vehicles is likely to change lanes. Processing unit 110 may additionally compare the curvature of the snail trail (associated with the leading vehicle) with the expected curvature of the road segment on which the leading vehicle is traveling. The expected curvature may be derived from map data (e.g., data from map database 160), road polynomials, the snail trails of other vehicles, prior knowledge of the road, and the like. If the difference between the curvature of the snail trail and the expected curvature of the road segment exceeds a predetermined threshold, processing unit 110 may determine that the leading vehicle is likely to change lanes.

[0113] In another embodiment, processing unit 110 may compare the current position of the leading vehicle with the look-ahead point (associated with vehicle 200) over a specified period of time (e.g., 0.5 to 1.5 seconds). If the distance between the current position of the leading vehicle and the look-ahead point varies during the specified time and the cumulative sum of the variation exceeds a predetermined threshold (e.g., 0.3 to 0.4 meters on a straight road, 0.7 to 0.8 meters on a moderately curved road, and 1.3 to 1.7 meters on a road with sharp curves), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the snail trail by comparing the lateral distance traveled along the trail with the expected curvature of the snail trail.After the calculation, the expected radius of curvature can be determined using the calculation: (δ. z 2 + δ x 2 ) / 2 / (δ x ), where δ xrepresents the lateral distance traveled, and δ2 represents the longitudinal distance traveled. If the difference between the lateral distance traveled and the expected curvature exceeds a predetermined threshold (e.g., 500 to 700 meters), the processing unit 110 may determine that the lead vehicle is likely to change lanes. In another embodiment, the processing unit 110 may analyze the position of the lead vehicle. If the position of the lead vehicle obscures a road polynomial (e.g., if the lead vehicle overlays the road polynomial), the processing unit 110 may determine that the lead vehicle is likely to change lanes.If the position of the leading vehicle is such that another vehicle is detected in front of the leading vehicle and the snail trails of the two vehicles are not parallel, the processing unit 110 may determine that the (closer) leading vehicle is likely to change lanes.

[0114] In step 584, the processing unit 110 may determine whether or not the lead vehicle 200 is changing lanes based on the analysis performed in step 582. For example, the processing unit 110 may make the determination based on a weighted average of the individual analyses performed in step 582. For example, in such a scheme, a decision by the processing unit 110 that the lead vehicle is likely to change lanes based on a particular type of analysis may be assigned the value "1" (and "0" to represent a determination that the lead vehicle is unlikely to change lanes). The various analyses performed in step 582 may be assigned different weights, and the disclosed embodiments are not limited to any particular combination of analyses and weights.Furthermore, in some embodiments, the analysis may use a trained system (e.g., a machine learning or deep learning system) that may estimate a future path ahead of a current location of a vehicle based, for example, on an image taken at the current location.

[0115] Fig. 6 is a flowchart illustrating an example method 600 for triggering one or more navigation responses based on stereo image analysis, in accordance with disclosed embodiments. In step 610, processing unit 110 may receive a first and a second plurality of images via data interface 128. For example, cameras included in image capture unit 120 (e.g., image capture devices 122 and 124 with fields of view 202 and 204) may capture a first and a second plurality of images of an area ahead of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first and second plurality of images via two or more data interfaces.The disclosed embodiments are not limited to any particular configurations of data interfaces or protocols.

[0116] In step 620, the processing unit 110 may execute the stereo image analysis module 404 to perform a stereo image analysis of the first and second plurality of images to create a 3D map of the road ahead of the vehicle and to detect features in the images, such as road markings, vehicles, pedestrians, traffic signs, highway exits, traffic lights, hazards, and the like. The stereo image analysis may be performed in a manner similar to that described in connection with Fig. 5A-5D, see above. For example, the processing unit 110 may execute the stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, pavement markings, traffic lights, hazards, etc.) in the first and second pluralities of images, filter out a subset of the candidate objects based on various criteria, and perform a multi-frame analysis, construct measurements, and determine a confidence level for the remaining candidate objects. In performing the above processes, the processing unit 110 may consider information from both the first and second pluralities of images, rather than just information from one set of images.For example, processing unit 110 may analyze the differences in pixel-level data (or other subsets of data from the two streams of acquired images) for an object that appears in both the first and second plurality of images. In a further process, processing unit 110 may estimate the position and / or velocity of a candidate object (e.g., relative to vehicle 200) by observing that the object appears in one of the plurality of images but not the other, or with respect to other differences that may exist with respect to objects that appear in the two image streams. For example, position, velocity, and / or acceleration relative to vehicle 200 may be determined based on trajectories, positions, motion characteristics, etc., of features associated with an object that appears in one or both image streams.

[0117] In step 630, the processing unit 110 may execute the navigation response module 408 to determine, based on the analysis performed in step 620 and the information provided above in connection with Fig. 4 to trigger one or more navigation responses in vehicle 200. Navigation responses may include, for example, a turn, a lane change, a change in acceleration, a change in speed, a deceleration, and the like. In some embodiments, processing unit 110 may use data resulting from the execution of speed and acceleration module 406 to trigger one or more navigation responses. Furthermore, multiple navigation responses may occur simultaneously, in sequence, or in any combination.

[0118] Fig. 7 is a flowchart illustrating an example process 700 for triggering one or more navigation responses based on an analysis of three sets of images consistent with the disclosed embodiments. In step 710, the processing unit 110 may receive a first, second, and third plurality of images via the data interface 128. For example, cameras included in the image acquisition unit 120 (such as image capture devices 122, 124, and 126 with fields of view 202, 204, and 206) may capture a first, second, and third plurality of images of an area in front of and / or to the side of the vehicle 200 and transmit them to the processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, the processing unit 110 may receive the first, second, and third plurality of images via three or more data interfaces.For example, each of the image capture devices 122, 124, 126 may have an associated data interface for transmitting data to the processing unit 110. The disclosed embodiments are not limited to any particular configurations of data interfaces or protocols.

[0119] In step 720, the processing unit 110 may analyze the first, second, and third plurality of images to identify features in the images, such as lane markings, vehicles, pedestrians, road signs, highway exits, traffic lights, road hazards, and the like. The analysis may be performed in a manner similar to the steps described in connection with Fig. 5A-5D and 6 above. For example, the processing unit 110 may perform monocular image analysis (e.g., via execution of the monocular image analysis module 402 and based on the information provided in connection with Fig. 5A-5D above) for each of the first, second, and third plurality of images. Alternatively, the processing unit 110 may perform a stereo image analysis (e.g., via execution of the stereo image analysis module 404 and based on the information obtained in connection with Fig. 6 above) for the first and second plurality of images, the second and third plurality of images, and / or the first and third plurality of images. The processed information corresponding to the analysis of the first, second, and / or third plurality of images may be combined. In some embodiments, the processing unit 110 may perform a combination of monocular and stereo image analyses. For example, the processing unit 110 may perform a monocular image analysis (e.g., via execution of the monocular image analysis module 402) for the first plurality of images and a stereo image analysis (e.g., via execution of the stereo image analysis module 404) for the second and third plurality of images.The configuration of the image capture devices 122, 124, and 126—including their respective locations and fields of view 202, 204, and 206—may influence the types of analyses performed on the first, second, and third pluralities of images. The disclosed embodiments are not limited to any particular configuration of the image capture devices 122, 124, and 126 or to the types of analyses performed on the first, second, and third pluralities of images.

[0120] In some embodiments, processing unit 110 may perform tests on system 100 based on the images captured and analyzed in steps 710 and 720. Such tests may provide an indication of the overall performance of system 100 for particular configurations of image capture devices 122, 124, and 126. For example, processing unit 110 may determine the proportion of "false hits" (e.g., cases where system 100 incorrectly detected the presence of a vehicle or pedestrian) and "misses."

[0121] In step 730, the processing unit 110 may trigger one or more navigation responses in the vehicle 200 based on information derived from two of the first, second, and third plurality of images. The selection of two of the first, second, and third plurality of images may depend on various factors, such as the number, types, and sizes of objects detected in each of the plurality of images. The processing unit 110 may also make the selection based on the image quality and resolution, the effective field of view reflected in the images, the number of frames captured, the extent to which one or more objects of interest actually appear in the frames (e.g., the percentage of frames in which an object appears, the proportion of the object that appears in each of those frames, etc.), and the like.

[0122] In some embodiments, the processing unit 110 may select information from two of the first, second, and third images by determining the extent to which the information from one image source matches the information from other image sources. For example, the processing unit 110 may combine the processed information from each of the image capture devices 122, 124, and 126 (whether through monocular analysis, stereo analysis, or any combination of both) and determine visual indicators (e.g., lane markings, a detected vehicle and its location and / or path, a detected traffic light, etc.) that are consistent across the images captured by each of the image capture devices 122, 124, and 126. The processing unit 110 may also exclude information that is inconsistent in the captured images (e.g.,a vehicle changing lanes, a lane model indicating a vehicle that is too close to the vehicle 200, etc.). Thus, based on the determinations of consistent and inconsistent information, the processing unit 110 may select information derived from two of the first, second, and third plurality of images.

[0123] The navigation responses may include, for example, a turn, a lane change, a change in acceleration, and the like. The processing unit 110 may determine the one or more navigation responses based on the analysis performed in step 720 and the information provided above in connection with Fig. 4. The processing unit 110 may also use data resulting from the execution of the speed and acceleration module 406 to trigger one or more navigation responses. In some embodiments, the processing unit 110 may trigger the one or more navigation responses based on a relative position, a relative speed, and / or a relative acceleration between the vehicle 200 and an object detected in one of the first, second, and third pluralities of images. Multiple navigation responses may occur simultaneously, in sequence, or in any combination. NAVIGATING LANE JOINS AND LANE SEPARATIONS

[0124] In one embodiment, an autonomous vehicle (e.g., vehicle 200) may encounter lanes of different types while navigating. For example, the lane the autonomous vehicle is traveling in may merge with an adjacent lane. During a lane merge, another vehicle may be traveling in an adjacent lane that is coming to an end, and that vehicle may merge in front of the autonomous vehicle. The autonomous vehicle may therefore slow down and / or adjust its course based on one or more characteristics of the other vehicle. For example, if the other vehicle overtakes quickly and without leaving much space between it, the autonomous vehicle may need to adjust its speed and / or change lanes. As another example, the lane the autonomous vehicle is traveling in may split to form an additional, adjacent lane, such as an exit lane.

[0125] During lane splitting, another vehicle may change lanes or exit a road ahead of the autonomous vehicle and no longer be positioned in the lane ahead of the autonomous vehicle. The autonomous vehicle can therefore maintain or increase its rate of acceleration to prevent the other vehicle from exiting the lane. However, if the vehicle is predicted to remain in the same lane as the autonomous vehicle beyond the split lane, the autonomous vehicle can maintain or decrease its rate of acceleration.

[0126] The disclosed embodiments include systems and methods for navigating an autonomous vehicle to take into account the above-mentioned lane change scenarios and possible maneuvers that other neighboring vehicles may perform in relation to the lane change.

[0127] For example, in one embodiment, a system for navigating a host vehicle (e.g., an autonomous vehicle) may include an image capture device and at least one processing device. The at least one processing device may be programmed to receive a plurality of images representing an environment of the host vehicle from the image capture device. The environment may include a road on which the host vehicle is traveling and a target vehicle. The target vehicle may be another vehicle located near the host vehicle and identified by the host vehicle's navigation system.

[0128] In a typical lane merging situation, the target vehicle can travel in a different lane of the road than the host vehicle. This means that the target vehicle can travel in an adjacent lane that converges with the lane in which the host vehicle is traveling. On the other hand, the target vehicle can typically travel in the same lane of the road as the host vehicle. This means that the target vehicle can travel in front of the host vehicle and in the same lane as the host vehicle.

[0129] The at least one processing device may be further programmed to analyze at least one of the plurality of images to identify the target vehicle and to analyze at least one of the plurality of images to identify at least one lane marker associated with the lane in which the target vehicle is traveling. A lane marker may, for example, be painted markings or indicators that mark a lane on a road, or a physical object that marks a lane on a road. Such physical objects may include barriers, dividers, median strips, etc.

[0130] The at least one processing device may be further programmed to detect one or more lane marker characteristics of the at least one identified lane marker and use the one or more detected lane marker characteristics to determine a type of the at least one identified lane marker. For example, a lane marker characteristic may indicate a lane marker characteristic that the system uses to determine the type of lane marker (e.g., whether it is a merge lane marker or a split lane marker).

[0131] Furthermore, the at least one processing device may be further programmed to determine at least one characteristic of the target vehicle.

[0132] A characteristic of the target vehicle may include, for example, the position of the target vehicle on the road. Based on the determined lane marking type and the determined characteristic of the target vehicle, the at least one processing device may be further programmed to determine a navigation action for the host vehicle. The navigation action may include changing or maintaining one or more of the steering, braking, or acceleration / deceleration functions of the host vehicle.

[0133] Fig. 8A is an illustration of exemplary lane merges on a roadway in accordance with the disclosed embodiments. As in Fig. As shown in Figure 8A, some lane merges may take the form of a "lane merge" 800. During a lane merge, the lane in which a host vehicle 200 is traveling, i.e., the "host" lane, may merge with an adjacent lane, requiring the host vehicle 200 and a target vehicle 802 to merge into the adjacent lane.

[0134] Another example of a lane merge is a "merge to" lane merge. In a lane merge 804, the "host" lane in which a host vehicle 200 is traveling may merge into an adjacent lane, as shown, so that the host vehicle "merges" into the adjacent lane and merges with the target vehicle 802 in the adjacent lane. If the adjacent lane is to the left of the host vehicle, as shown, the lane merge may be a "left merge," while if the adjacent lane is to the right of the host vehicle, the lane merge may be a "right merge."

[0135] Another example of a lane merge is a "merge from" lane merge 806. In a "lane merge from," an adjacent lane may merge into the "host" lane in which a host vehicle 200 is traveling, such that a target vehicle 802 in the adjacent lane merges "from" the adjacent lane into the "host" lane in which the host vehicle 200 is traveling. As shown, if the adjacent lane is to the right of the host vehicle 200, such that the target vehicle 802 is "merging from the right," the lane merge may be a "merge from the right," whereas if the adjacent lane is to the left of the host vehicle 200, such that the target vehicle 802 is "merging from the left," the lane merge may be a "merge from the left."

[0136] Other lane changes may take the form of lane separation. Fig. 8B illustrates exemplary lane separations on a roadway in accordance with the disclosed embodiments. For example, in a "split" lane split 808, a "host" lane in which a host vehicle 200 is traveling may split into two lanes, such that the host vehicle 200 selects a lane in which to travel. In some cases of a "split" lane split, a target vehicle (in the example of a "split" lane split in Fig. 8B not shown) may be traveling ahead of the host vehicle. The target vehicle can also select a lane in which to travel.

[0137] Another example of a lane separation is an "open" lane 810. In an "open" lane separation 810, a "host" lane in which the host vehicle 200 is traveling may remain unchanged, but a new lane may appear adjacent to the "host" lane in which the host vehicle 200 is traveling. The new adjacent lane may be, for example, an exit lane. If the new adjacent lane is to the right of the "host" lane in which the host vehicle 200 is traveling, the lane separation may be an "open right" lane separation, while the new adjacent lane is to the left of the "host" lane in which the host vehicle 200 is traveling, the lane separation may be an "open left" lane separation. In some cases of an "open" lane separation, a target vehicle (in the example of an "open" lane separation in Fig. 8B not shown) in front of the host vehicle. The target vehicle can continue in the same lane or enter the new lane.

[0138] Another example of a lane split is the "widening" of lane 812. In a "lane widening," the "host" lane in which the host vehicle 200 is traveling may widen into a new, adjacent lane (lane split 812). The new adjacent lane may be, for example, an exit lane. If the new adjacent lane is to the right of the "host" lane in which the host vehicle 200 is traveling, the lane split may be a "right lane widening," while if the new adjacent lane is to the left of the "host" lane in which the host vehicle 200 is traveling, the lane split may be a "left lane widening." In some cases of an "open" lane split, a target vehicle 802 may be traveling in front of the host vehicle. The target vehicle 802 may continue in the same lane or enter the new lane.

[0139] Fig. 9 is a flowchart illustrating an exemplary process 900 for determining one or more navigation actions based on a lane change analysis in accordance with the disclosed embodiments. The process 900 may be performed, for example, by the processing unit 110 described above. For illustrative purposes, the process 900 is described with reference to Fig. 10A, which is an illustration of an autonomous vehicle (e.g., vehicle 200) in plan view of a roadway with lane merging in accordance with the disclosed embodiments, and with reference to Fig. 10B, which provides an exemplary illustration of a lane change analysis for lane merging in accordance with the disclosed embodiments. In the following discussion, the autonomous vehicle is referred to as the host vehicle (e.g., vehicle 200) and may include one or more components of system 100, as described above in connection with Fig. 1 described.

[0140] In step 902, the processing unit 110 may receive a plurality of images from an image capture device that represent an environment of the host vehicle. The environment includes a road on which the host vehicle is traveling. As shown in Fig. As shown in Figure 10A, the image capture device 122 may capture at least one image representing an environment 1000 of the host vehicle 200. The environment 1000 may include a road 1002 on which the host vehicle 200 is traveling.

[0141] The image capture device 122 may take any of the forms described above for the image capture device 122, the image capture device 124, and / or the image capture device 126. While the host vehicle 200 is illustrated with an image capture device 122 in a particular location, it should be understood that the host vehicle 200 may also include additional image capture devices 122 and / or additional or alternative locations for the image capture devices 122 (e.g., as illustrated in any of the preceding figures) that are not shown in Fig. 10A are shown.

[0142] As shown, the environment 1000 includes a target vehicle 802 in addition to the road 1002. Furthermore, the host vehicle 200 travels in one lane of the road 1002, and the target vehicle 802 travels in a different lane of the road 1002 than the host vehicle 200. While a single target vehicle 802 is shown at a particular location on the road 1002, it is understood that additional target vehicles 802 and / or additional or alternative locations for target vehicles 802 are possible. Although the road 1002 is shown as including two lanes in a particular configuration, it is understood that the road 1002 may include any number of lanes in any configuration.

[0143] Road 1002 includes lane markings 1004A-1004D. While lane markings 1004A-1004D are illustrated as two-dimensional markings (e.g., paint, decals, etc.) on road 1002, in some embodiments, lane markings 1004A-1004D may also be three-dimensional markings such as barriers, dividers, etc. Other lane markings 1004A-1004D are also possible. And while lane markings 1004A-1004D are shown in particular locations and having particular configurations, it should be understood that more or fewer lane markings 1004A-1004D and / or lane markings 1004A-1004D with alternative locations and / or configurations may be contemplated.

[0144] As previously mentioned, one or more image capture devices (e.g., image capture device 122) may capture at least one image representing the environment 1000. In some embodiments, the image(s) captured by the image capture device 122 may include a representation of the road 1002, the target vehicle 802, and / or one or more of the lane markers 1004A-1004D on the road 1002. The image(s) may, for example, be in the form of grayscale or color images. In some embodiments, the image(s) are not necessarily full-resolution images. Alternatively or additionally, the resolution within an image may vary as needed to identify and / or label lane markers, as described below.For example, a horizontal stripe encompassing the road area and the horizon may have a higher resolution in an image than horizontal stripes representing the sky and / or the hood of the host vehicle. As another example, horizontal stripes of an image representing the horizon may have a higher resolution than horizontal stripes representing the road closer to the host vehicle, because lane markings closer to the horizon may be smaller and more difficult to detect. In some embodiments, portions of an image may be subsampled or upsampled as desired to adjust resolution.

[0145] In order to Fig. 9, in step 904, the processing unit 110 analyzes at least one of the plurality of images to identify a target vehicle traveling in a lane of the road that is different from the lane in which the host vehicle is traveling. As in Fig. 10A, the images captured by the image acquisition unit 122 include, for example, a representation of the target vehicle 802 traveling in a lane of the road 1002 that is different from the lane in which the host vehicle 200 is traveling. The lane in which the target vehicle 802 is traveling includes a neighboring and / or adjacent lane to the lane in which the host vehicle 200 is traveling. The processing unit 110 analyzes one or more of the captured images to identify the target vehicle 802, for example, using the monocular image analysis module 402 and / or the stereo image analysis module 404 described above in connection with Fig. 4 were described.

[0146] In order to Fig. 9, the processing unit 110 analyzes at least one of the plurality of images in step 906 to identify at least one lane marking associated with the lane in which the target vehicle is traveling. As shown in Fig. 10A, the images captured by unit 122 include, for example, the representation of lane markings 1004C and 1004D on road 1002. Processing unit 110 analyzes the images to identify lane markings 1004C and 1004D, for example, using monocular image analysis module 402 and / or stereo image analysis module 404 described above in connection with Fig. 4. Alternatively or additionally, in some embodiments, processing unit 110 may identify lane markers 1004C and 1004D in a manner similar to that described in Stein, U.S. Patent No. 7,151,996, the contents of which are incorporated by reference in their entirety. In some embodiments, processing unit 110 may identify one or both lane markers 1004A and 1004B in any of the ways described above.

[0147] In order to Fig. 9, the processing unit 110 may, in step 908, detect one or more lane marking properties of the at least one identified marking. The detected lane marking property(ies) may include, for example, the distance of the at least one identified lane marking to a reference point. The reference point may include, for example, a sign, a lamppost, a traffic signal, a curb, or another feature of the road. Alternatively or additionally, the reference point may also include another landmark (e.g., a tree, a building, a bridge, an overpass, a parked vehicle, a body of water, etc.). Furthermore, the reference point may include another lane marking on the road. In Fig. 10A, the determined properties of the lane marking(s) may include, for example, a distance between lane markings 1004A and 1004C or between lane markings 1004B and 1004C. As another example, the determined properties of the lane marking(s) may include an intersection of the at least one identified lane marking with at least one other lane marking of the road. As shown in Fig. 10A, the detected lane marking property(ies) may include an intersection of lane markings 1004B and 1004C. As another example, the detected lane marking property(ies) may include a change in the lateral distance between the at least one identified lane marking and at least one other lane marking of the road 1002. In Fig. 10A, the detected lane marking property(ies) may include, for example, a change in the lateral spacing between lane markings 1004A and 1004C or between lane markings 1004B and 1004C.

[0148] In order to Fig. 9, in step 910, the processing unit 110 may use the at least one determined property of the lane marking to determine a type of the at least one identified lane marking. In some embodiments, such as the one in Fig. 10A, the lane marker type may be a merge lane. A lane merge may include a "lane merge," a "lane merge to," or a "lane merge from" lane merge, as described above in connection with Fig. 8A. In some embodiments, the lane marking type may also be a split lane, such as a “split” split lane, an “open” split lane, or an “extended” split lane, as described above in connection with Fig. 8B and below in connection with Fig. 11 discussed.

[0149] In embodiments where the determined property(ies) of the lane marking include, for example, a distance of the at least one identified lane marking to a reference point, such as another marking on the road, the use of the determined property(ies) of the lane marking to determine a type of the identified lane marking(s) may include determining whether the distance between the identified lane marking(s) and the reference point increases, decreases, or remains constant with increasing distance from the host vehicle (e.g., over a distance along the road in front of the host vehicle). Fig. For example, in Figure 10A, it can be seen that if the determined lane marker property(ies) include a distance between lane markers 1004A and 1004C and / or between lane markers 1004B and 1004C, a decreasing distance between the lane markers may indicate that the lane marker type for lane 1004C is a lane merge. Conversely, if the distance between lane markers 1004A and 1004C and / or between lane markers 1004B and 1004C increases, processing unit 110 may determine that the lane marker type for lane marker 1004C is a split lane.

[0150] Alternatively or additionally, in embodiments where the determined property(ies) of the lane marking(s) include, for example, an intersection of the identified lane marking(s) with at least one other lane marking on the road, using the determined property(ies) of the lane marking(s) to determine a type of the identified lane marking(s) may include determining that the intersection indicates that the type of lane marking is a lane merge. For example, see Fig. 10A, if the determined lane marker property(ies) include an intersection of lane markers 1004B and 1004C, processing unit 110 may determine that the lane marker type for lane marker 1004C is a lane merge.

[0151] Still alternatively or additionally, if the determined lane marking property(ies) include a change in the lateral distance between the at least one identified lane marking and at least one other road marking on the road 1002, using the determined lane marking property(ies) to determine a type of the identified lane marking(s) may include determining whether the lateral distance between the identified lane marking(s) and the other lane marking(s) increases, decreases, or remains constant with increasing distance from the host vehicle.

[0152] For example, determining a lateral distance between lane markings over a distance along a roadway in front of the host vehicle may include comprising a lateral distance A between a first lane marking and a second lane marking at a reference point A in front of the host vehicle and determining a lateral distance B between the first lane marking and the second lane marking at a reference point B in front of the reference point A. Comparing the lateral distance A and the lateral distance B may indicate whether the lateral distance between the lane markings is increasing or decreasing. When the lateral distance A is greater than the lateral distance B, the lateral distances between the first lane marking and the second lane markings may be decreasing.A decreasing lateral distance between lane markings with increasing distance in front of the host vehicle may indicate that a lane marking is a merge lane marking. If the lateral distance A is smaller than the lateral distance B, the lateral distances between the first lane marking and the second lane marking may increase. An increasing lateral distance between lane markings with increasing distance in front of the host vehicle may indicate that a lane marking is a split lane marking. Other comparisons are conceivable and consistent with the disclosed embodiments. Determining the types of lane markings may, for example, involve calculating the difference between the lateral distance A and the lateral distance B, or vice versa. For example, if the difference between the lateral distance A and the lateral distance B (i.e.If the difference between lateral distance A and lateral distance B (i.e., A minus B) is a positive value, then the lateral distances between lane markings may decrease over a distance in front of the host vehicle, which may indicate a merging lane marking. However, if the difference between lateral distance A and lateral distance B (i.e., A minus B) is a negative value, then the lateral distances between lane markings may increase over a distance in front of the host vehicle, which may indicate that a lane marking is a split lane marking.

[0153] For example, if Fig. 10A, the determined lane marking property(ies) include a lateral distance between lane markings 1004A and 1004C and / or between lane markings 1004B and 1004C, a decreasing lateral distance between the lane markings over a distance in front of the host vehicle may indicate that the lane marking type for lane marking 1004C is a merge lane. Conversely, if the lateral distance between lane markings 1004A and 1004C and / or between lane markings 1004B and 1004C increases over a distance in front of the host vehicle, processing unit 110 may determine that the lane marking type for lane marking 1004C is a split lane.

[0154] Other characteristics of the lane marker are conceivable and consistent with the disclosed embodiments. For example, a characteristic of the lane marker may include one or more characteristics such as the size, shape, and color of the lane marker. Further, in some embodiments, a characteristic of the lane marker may include a positional relationship between a lane marker and another lane marker. For example, a lane marker may be adjacent to another lane marker (e.g., adjacent to or connected to it), or the lane marker may be oriented in a particular direction (e.g., at a particular angle) with respect to another lane marker.

[0155] In order to Fig. Returning to Figure 9, in step 912, the processing unit 110 determines at least one characteristic of the target vehicle. To determine the characteristic(s) of the target vehicle, analyze the plurality of images as described above in connection with steps 904 and 906. In some embodiments, determining the characteristic(s) of the target vehicle may include analyzing data from one or more sensors and / or devices of the host vehicle (e.g., an image sensor, lidar, radar, a GPS device, a speed sensor, an accelerometer, a suspension sensor, an acoustic sensor, etc.).

[0156] In some embodiments, the particular characteristic(s) of the target vehicle may include a position of the target vehicle on the road ahead of the host vehicle. For example, with respect to Fig. 10A, the particular characteristic(s) of the target vehicle 802 may include a position of the target vehicle 802 on the road 1002. The position may be a position relative to the host vehicle 200, one or more of the lane markers 1004A-1004D, and / or another connection point on the road 1002, or position data (e.g., GPS position data), as described above in connection with Fig. 1. The determined characteristic(s) of the target vehicle include(s) a detected lateral movement of the target vehicle relative to the identified lane marking(s). With reference to Fig. 10A, the particular characteristic(s) of the target vehicle 802 includes a detected lateral movement of the target vehicle 802 relative to the lane marker 1004C. In some embodiments, the particular characteristic(s) of the target vehicle may include a speed of the target vehicle relative to the host vehicle. For example, with respect to Fig. 10A, the determined characteristics of the target vehicle 802 may include a speed of the target vehicle 802 relative to the host vehicle 200. In some embodiments, the determined characteristic(s) of the target vehicle may include a predicted trajectory of the target vehicle. For example, with respect to Fig. 10A, the determined characteristic(s) of the target vehicle 802 may include a predicted trajectory of the target vehicle 802. The processing unit 110 may predict the trajectory of the target vehicle 802 based on an analysis of at least two of the images captured in step 902.

[0157] In order to Fig. 9, the processing unit 110 may determine a navigation action for the host vehicle in step 914 based on the determined type of lane marking and the determined property of the target vehicle. The navigation action may include changing or maintaining one or more of the steering, braking, or acceleration / deceleration functions of the host vehicle. In some embodiments, the navigation action may be performed using one or more of the methods described above in connection with Fig. 2F and Fig. 4, the navigation action may be performed by the modules throttle system 220, braking system 230, steering system 240, speed and acceleration module 406, and navigation response module 408. The navigation action may, for example, be increasing, maintaining, or decreasing the rate of acceleration of the host vehicle. Another example is steering the host vehicle, such as steering left or right.

[0158] The determined navigation action for the host vehicle may be based on the determined type of lane marking. For example, if the determined type of lane marking is a lane merge, the navigation action for the host vehicle may be determined to facilitate a safe merge with the target vehicle. This may include, for example, accelerating or decelerating the host vehicle to avoid a collision with the target vehicle. In some cases, the navigation action may include steering the host vehicle into an adjacent lane to safely merge with the target vehicle. As another example, if the determined type of lane marking is a lane split, the navigation action for the host vehicle may be determined to facilitate the host vehicle's travel in the corresponding lane. This may include, for example, steering the host vehicle to remain in the corresponding lane.

[0159] The determined navigation action for the host vehicle is based on the determined characteristic of the target vehicle. For example, if the determined characteristic(s) of the target vehicle include a position of the target vehicle on the road ahead of the host vehicle, the navigation action for the host vehicle may be determined to avoid a collision with the target vehicle, e.g., by accelerating or decelerating the host vehicle. As another example, if the determined characteristic(s) of the target vehicle include a detected lateral movement of the target vehicle with respect to the identified lane marker(s), the navigation action for the host vehicle may be determined to facilitate safe lane merging with the target vehicle. Fig. 10A, for example, the processing unit 110 uses the detected lateral movement of the target vehicle 802 relative to the lane marker 1004C to determine whether the target vehicle 802 is moving at an appropriate speed and / or maintaining a safe distance from the host vehicle 200. For example, if the target vehicle 802 maintains a position parallel or substantially parallel to the lane marker 1004C or moves at an angle of approximately 90 degrees across the lane marker 1004C, it is likely that the target vehicle 802 is moving at an appropriate speed and a sufficient distance from the host vehicle 200. However, if the target vehicle 802 is moving such that its trajectory intersects the lane marker 1004C at an angle (e.g., at a small angle of less than 90 degrees), then the target vehicle 802 is likely to move quickly and merge close to the host vehicle 200.The navigation action may be a change in the acceleration of the host vehicle 200 to compensate for the lateral movement of the target vehicle 802. As another example, if the particular characteristic(s) of the target vehicle include a speed of the target vehicle relative to the host vehicle, the navigation action for the host vehicle may include maintaining a safe distance from the target vehicle, e.g., by accelerating or decelerating. As another example, if the particular characteristic(s) of the target vehicle include a predicted trajectory of the target vehicle, the navigation action for the host vehicle may include avoiding the trajectory of the target vehicle, e.g., by steering and / or accelerating or decelerating the host vehicle.

[0160] As another example, if the particular type of lane marking is a lane separation and a target vehicle is in front of the host vehicle but in front of the lane separation, the navigation action for the host vehicle may include acceleration or deceleration of the host vehicle, depending on the particular characteristic of the target vehicle. For example, the particular characteristic of the target vehicle may indicate that the target vehicle is (or will be) changing lanes or leaving (or will be leaving) the road and can no longer be positioned in front of the host vehicle. The host vehicle may therefore maintain or increase its acceleration rate in anticipation of the target vehicle leaving the lane.As another example, if the determined characteristic of the target vehicle indicates that the target vehicle remains (or will remain) in the same lane as the host vehicle after lane separation, the host vehicle may maintain or decrease its acceleration rate. In such embodiments, determining the navigation action for the host vehicle may include considering one or more predicted navigation actions of the target vehicle, as described below.

[0161] For example, in addition to or as part of determining a navigation action for the host vehicle, processing unit 110 may predict a navigation action of the target vehicle. Processing unit 110 may predict the navigation action of the target vehicle based on the type of lane in which the target vehicle is traveling. For example, if the target vehicle is traveling in one lane, processing unit 110 may predict that the target vehicle will change lanes. As another example, if the target vehicle is traveling in a divided lane, processing unit 110 may predict that the target vehicle will exit the roadway.

[0162] As another example, referring to Fig. 10A, the processing unit 110 may identify a position of the target vehicle 802 relative to the lane marker 1004C marking a lane merge to determine whether the target vehicle 802 is entering the lane in which the host vehicle 200 is traveling. For example, if the lateral distance between the target vehicle 802 and the lane marker 1004C is constant or decreasing, the processing unit 110 may determine that the target vehicle 802 is merging into the lane in which the host vehicle 200 is traveling. In this manner, the processing unit 110 may determine whether the target vehicle 802 is tracking the lane marker 1004C and, accordingly, whether the target vehicle 802 is merging into the lane in which the host vehicle 200 is traveling even before the target vehicle 802 crosses the lane marker 1004B into the exit lane.In some embodiments, the predicted navigation action of the target vehicle may be determined based on a predicted trajectory of the target vehicle. For example, if the target vehicle 802 is predicted to enter the lane in which the host vehicle 200 is located, the host vehicle 200 may decelerate, accelerate, change lanes, or use the horn to facilitate a safe merge. Such predicted navigation actions of the target vehicle may be based on the analysis of at least two images captured by an image capture device of the host vehicle. In still other embodiments, such predictive navigation actions may be based on the analysis of data from one or more sensors and / or devices of the host vehicle (e.g.,an image sensor, lidar, radar, a GPS device, a speed sensor, an accelerometer, a suspension sensor, an acoustic sensor, etc.).

[0163] In some embodiments, in addition to identifying lane markers 1004A-1004D, processing unit 110 may label lane markers 1004A-1004D. For example, processing unit 110 may identify lane markers 1004A-1004D in the images acquired by image processing unit 122 and generate a label for each identified lane marker 1004A-1004D. The labels may, for example, take the form of a binary map. Alternatively, the labels may also take the form of a data structure, such as a list of image segment coordinates.

[0164] In some embodiments, generating the labels may consist of feeding the image(s) to a trained system that recognizes the shape of one of the labels described above in connection with Fig. 4 and / or a convolutional neural network. The neural network may be trained using labels and / or lane marker identifiers associated with one or more images. Labeled data or images associated with labels and / or lane marker identifiers may be received by the neural network from a variety of resources, e.g., one or more databases. The one or more databases may be located locally (e.g., included in system 100) and / or remotely (e.g., via a network such as a cellular network and / or the Internet, etc.) and may be received, e.g., via a wireless transceiver 172 of system 100. The labeling of the stored images is performed, e.g., by storing a label (e.g., a lane merge or lane split) associated with an identified lane marker.In some embodiments, not every identified lane marker may be labeled. Rather, the labeling may be limited to those lane markers relevant for detecting a lane change and determining a lane type, as described below. Finally, the neural network could be capable of automatically identifying and labeling the lane markers.

[0165] In some embodiments, the neural network may output labeled images in which pixels of the images representing lane markers 1004A-1004D are labeled as part of a lane marker. In some embodiments, only the pixels representing lane markers 1004A-1004D may be labeled and / or associated with an identifier for the type of lane marker. Alternatively, in some embodiments, every pixel may be labeled, even those that do not represent lane markers 1004A-1004D, and the labeled image may be "anded" with the binary map to isolate labels for lane markers 1004A-1004D.

[0166] In some embodiments, the labeled images are not necessarily full-resolution images. Alternatively or additionally, the labeled images may not have the same resolution as the images captured by the image acquisition unit 122 and / or the binary map. In some embodiments, the images captured by the image acquisition unit 122 may have a higher resolution than the labeled images, as the higher resolution may assist the processing unit 110 in labeling the lane markers, but a lower resolution may be sufficient to display the labeled lane markers. For example, the identified lane markers may be downsampled to a lower resolution for the labeled images and / or the labeled lane markers may be upsampled to a higher resolution for the identified lane markers.

[0167] Example markings of tracks are in Fig. 10B, which illustrates sections of road 1002 captured in a first image 1006. For example, processing unit 110 may identify lane markings 1004C and 1004D in the first image 1006. Alternatively or additionally, processing unit 110 may mark lane markings 1004A and 1004B. Once identified, processing unit 110 may label lane markings 1004A, 1004B, 1004C, and 1004D and / or generate lane marking identifiers for one or more of the lane markings.

[0168] In some embodiments, lane markers may be labeled as "normal" lane markers or as "lane change" lane markers. For example, lane markers 1004A and 1004D in the first image 1006 may be labeled as "normal" lane markers, while lane markers 1004B and 1004C are labeled as "lane change" lane markers. In some embodiments, "lane change" lane markers may be stored as known landmarks, as described above in connection with Fig. 1. For example, the coarse location of a lane merge or lane split may be determined based on the intersection and / or the relative location of one or more lane change landmarks. Such lane change landmarks may enhance localization technology, such as the REM™ used in mapping. In some embodiments, the localization technology may rely on lane change landmarks identified by a large number of vehicles traveling on the same roadway to determine which lane markers can be merged and which cannot.For example, based on the identified "lane change" lane markers, the localization technology can detect that numerous lane marker identifiers before a split can be merged into a consistent model, while numerous lane marker identifiers after the split cannot. As another example, based on the identified "lane change" lane markers, the localization technology can detect that numerous lane marker identifiers after a lane merge can be merged into a consistent model, while numerous lane marker identifiers before the lane split cannot.

[0169] In some embodiments, processing unit 110 may further determine, for one or more identified lane markers, the relationship of the lane marker to host vehicle 200. For example, lane marker 1004A may be labeled a "host left" lane marker, indicating the left side of a "host" lane in which host vehicle 200 is traveling. As another example, lane marker 1004B may be labeled a "host right" lane marker, indicating the right side of a "host" lane in which host vehicle 200 is located. Lane marker 1004D may be labeled a "next right" lane marker, marking the right side of a "next right" lane to host vehicle 200.

[0170] As in Fig. 10B, the processing unit 110 may additionally identify and mark lane markings and / or generate lane marking identifiers for lane markings in a second image 1008. The second image 1008 may be captured at a later time and / or at a location farther along the road 1002 than the first image 1006. In some embodiments, the first image 1006 and the second image 1008 may enable the processing unit 110 to identify and mark lane markings at a distance apart on the road 1002. The distance between the images may be, for example, between 25 and 75 meters. Other distances are also possible.

[0171] As shown, lane markers 1004A and 1004D may be marked in the second image 1008. The processing unit 110 may mark lane marker 1004A with a "Host Left" lane marker, which in turn indicates the left side of the lane in which the host vehicle 200 is traveling. However, lane marker 1004D may be marked in the second image 1008 as a "Host Right" lane marker, which indicates the right side of the lane in which the host vehicle 200 is traveling.

[0172] In some embodiments, determining the type of identified lane marker(s) in step 910 may include determining a change in the labels of the lane markers and / or the identifiers of the lane markers between the images. Such a determination may allow the processing unit 110 to further determine what type of lane merge (e.g., "merge," "merge to," or "merge from") or what type of lane split (e.g., "lane split," "lane split open," or "lane widening") is occurring on the road 1002. For example, with respect to Fig. 10B, the processing unit 110 may determine that the first image 1006 includes a “host left” lane marker (shown as lane marker 1004A and labeled “HL”), a “host right” lane marker (shown as lane marker 1004B and labeled “HR”), and a “next right lane” lane marker (shown as lane marker 1004 and labeled “NRR”), as described above. The processing unit 110 may further determine that a second image 1008 still includes a "host left" lane marker (labeled "HL") and a "host right" lane marker (labeled "HR"), but not the "next right lane" lane marker from the first image 1006. Based on these labels, the processing unit 110 may determine that the "next right" lane, marked in part by the "next right lane" lane marker in the first image 1006, merges into the "host" lane in which the host vehicle 200 is traveling.Accordingly, the processing unit 110 may determine that the target vehicle 802 is traveling on a “lane merge from,” that is, the host vehicle 200 may prepare for the target vehicle 802 to merge from its lane (the “next right” lane) into the “host” lane in which the host vehicle 200 is located.

[0173] As another example, processing unit 110 may determine that first image 1006 includes a "host left" lane marker, a "host right" lane marker, and a "next right" lane marker. Processing unit 110 may further determine that, in a second image 1008, the "host right" lane marker has become a "host left" lane marker and the "next right" lane marker has become a "host right" lane marker. The second image 1008 may not include a "host left" lane marker. Based on these markings, processing unit 110 may determine that the "host" lane, marked by the "host right" and "host left" lane markers in first image 1006, transitions into the "next right" lane, marked in part by the "next right" lane marker, in which target vehicle 802 is traveling.Accordingly, the processing unit 110 may determine that the target vehicle 802 is traveling in a “lane merge” lane, meaning that the host vehicle 200 is preparing to transition from the “host” lane to the “next right” lane in which the target vehicle 802 is traveling.

[0174] Although Fig. 10B shows a target vehicle 802, it should be understood that in some embodiments, the labels of the lane markers may be used to determine a navigation action for the host vehicle 200 even when no target vehicle is present. For example, in some embodiments, a neural network fed with labels as described above may be configured to detect a lane merge, as shown in Fig. 10B, based on designated lane markers, and the host vehicle 200 may take a navigation action to enable lane merging.

[0175] Fig. 11 is a flowchart illustrating another exemplary process 1100 for determining one or more navigation actions based on a lane change analysis in accordance with the disclosed embodiments. The process 1100 may be performed, for example, by the processing unit 110 described above. For illustrative purposes, the process 1100 is described with reference to Fig. 12A, which illustrates a top view of an autonomous vehicle (e.g., vehicle 200) on a roadway with a lane separation in accordance with the disclosed embodiments, and with reference to Fig. 12B, which provides an exemplary illustration of a lane change analysis for lane separation in accordance with the disclosed embodiments. In the following discussion, the autonomous vehicle is referred to as the host vehicle (e.g., vehicle 200) and may include one or more components of system 100, as described above in connection with Fig. 1 described.

[0176] In step 1102, the processing unit 110 may receive a plurality of images representing an environment of the host vehicle from an image capture device. The environment includes a road on which the host vehicle is traveling. As shown in Fig. 12A, for example, the image capture device 122 may capture a plurality of images representing an environment 1200 of the host vehicle 200. The environment 1200 may include a road 1202 on which the host vehicle 200 is traveling, and the road 1202 may include lane markings 1204A-1204D. The image capture device 122, the road 1202, and the lane markings 1204A-1204D may take any of the forms described above for the image capture device 122, the road 1202, and the lane markings 1204A-1204D in connection with Fig. 10 were described.

[0177] As previously mentioned, one or more image capture devices (e.g., image capture device 122) may capture a plurality of images representing the environment 1200. In some embodiments, the images captured by image capture device 122 may include a representation of road 1202 and / or one or more of the lane markings 1204A-1204D on road 1202. The images may take any of the forms described above for the images associated with Fig. 9 were described.

[0178] In step 1104, the processing unit 110 may analyze at least one of the plurality of images to identify a target vehicle traveling in a lane of the road on which the host vehicle is traveling. That is, in a split lane scenario, the target vehicle may be traveling in front of the host vehicle and in the same lane as the host vehicle prior to the split. As shown in Fig. 12A, the images captured by the image acquisition unit 122 may, for example, include a representation of the target vehicle 802 traveling in front of the host vehicle 100 in a lane of the road 1202 on which the host vehicle 200 is traveling. The processing unit 110 may analyze the image(s) to identify the target vehicle 802, for example, using the monocular image analysis module 402 and / or the stereo image analysis module 404 described above in connection with Fig. 4 were described.

[0179] In step 1106, the processing unit 110 may analyze at least one of the plurality of images to identify a plurality of lane markers associated with the lane. As shown in Fig. 12A, the image(s) captured by unit 122 may include a representation of lane markings 1204A and 1204B on road 1202. In some embodiments, the image(s) may additionally include lane markings 1204C and 1204D. Processing unit 110 may analyze the image(s) to identify lane markings 1204A-1204D, for example, using monocular image analysis module 402 and / or stereo image analysis module 404 described above in connection with Fig. 4 and / or in any other manner described above.

[0180] In step 1108, the processing unit 110 may detect one or more properties of the at least one identified lane marking. The detected property(ies) of the lane marking may include, for example, the distance of the at least one identified lane marking to a reference point, such as another marking on the road. For example, the Fig. 12A may include the distance between lane markings 1204A and 1204C or between lane markings 1204B and 1204D. As another example, the detected lane marking properties may include a splitting of the at least one identified lane marking into another lane marking. In Fig. 12A, the detected property(ies) of the lane marking may include, for example, a splitting of lane marking 1204B into lane marking 1204C. As another example, the detected properties of the lane marking(s) may include a change in the lateral distance between the at least one identified lane marking and at least one other lane marking of the road. For example, with respect to Fig. 10A, the detected characteristics of the lane marking(s) may include a change in the lateral distance between lane markings 1204A and 1204C or between lane markings 1204B and 1204C over a distance in front of the host vehicle. Such changes in lateral distances may be determined as described above, for example, in conjunction with Fig. 9.

[0181] In step 1110, the processing unit 110 may use the at least one detected property of the lane marking to determine a type of the at least one identified lane marking. Fig. 12A, the particular type of lane marking may be a split lane. A split lane may include a "split" split lane, an "open" split lane, or an "extended" split lane, as described above in connection with Fig. 8B. In some cases, the divided lane may include an exit lane (e.g., an exit lane of a road or a highway such as an interstate).

[0182] In embodiments where the determined property(ies) of the lane marking include, for example, a distance of the at least one identified lane marking to a reference point, such as another marking on the road, the use of the determined property(ies) of the lane marking to determine a type of the identified lane marking(s) may include determining whether the distance between the identified lane marking(s) and the reference point increases, decreases, or remains constant with increasing distance from the host vehicle (e.g., over a distance along the road in front of the host vehicle). For example, with respect to Fig. 12A, if the observed lane marking property(ies) include a spacing between lane markings 1204A and 1204C and / or between lane markings 1204B and 1204D, an increasing spacing between the lane markings may indicate that the lane marking type is a split lane.

[0183] Alternatively or additionally, in some embodiments, a detected property of the lane marking may include identifying a split of the detected lane marking(s) into another lane marking. For example, with respect to Fig. 12A, if a determined characteristic of the lane marker includes a split of the lane marker 1204B into a lane marker 1204B and a lane marker 1204C, the processing unit 110 may determine that the type of the lane marker is a split lane.

[0184] Still alternatively or additionally, if the determined lane marking property(ies) comprise a change in the lateral distance between the at least one identified lane marking and at least one other road marking, using the determined lane marking property(ies) to determine a type of the identified lane marking(s) may comprise determining whether the lateral distance between the identified lane marking(s) and the other lane marking(s) increases, decreases, or remains constant with increasing distance from the host vehicle (e.g., over a distance along the road in front of the host vehicle). For example, with respect to Fig. 12A, if the determined lane marking property(ies) include a lateral distance between lane markings 1204A and 1204C and / or between lane markings 1204B and 1204D, an increasing lateral distance between the lane markings may indicate that the lane marking type is a split lane. Such changes in the lateral distances between lane markings over a distance along the road in front of the host vehicle may be determined as described above, for example, in conjunction with Fig. 9.

[0185] In order to Fig. 11, processing unit 110 may determine at least one characteristic of the target vehicle in step 1112. In some embodiments, determining the characteristic(s) of the target vehicle may include analyzing the plurality of images as described above in connection with steps 1104 and 1106. The determined characteristic(s) of the target vehicle may take any of the forms described above in connection with step 912.

[0186] In step 1114, the processing unit 110 may determine a navigation action for the host vehicle based on the determined type of lane marking and the determined property of the target vehicle. The navigation action may take any of the forms described above, for example, also in connection with step 914 in Fig. 9.

[0187] While step 1114 provides for determining a navigation action for the host vehicle based on both the determined type of lane marking and the determined characteristic of the target vehicle, it should be understood that in some embodiments, the navigation action may be determined based only on the determined type of lane marking or only on the determined characteristic of the target vehicle. Fig. For example, while Figure 12A shows the target vehicle 802, in some embodiments, no target vehicle 802 may be present and the navigation action may be determined based only on the detected lane marker type.

[0188] For example, if the detected property(ies) of the target vehicle include a detected lateral movement of the target vehicle with respect to the identified lane marker(s), the navigation action for the host vehicle may be determined to enable the target vehicle to exit the road. For instance, referring to Fig. 12A, processing unit 110 may use the detected lateral motion of target vehicle 802 relative to lane mark 1204B to determine whether target vehicle 802 will stay in the lane in which host vehicle 802 is traveling or will exit road 1202. For example, if target vehicle 802 continues to travel parallel to lane marking 1204B while traveling, target vehicle 802 is more likely to stay in the lane in which host vehicle 200 is traveling, and host vehicle 200 can maintain a safe following distance and not accelerate. On the other hand, if the lateral distance indicates a slight movement of the target vehicle away from lane marking 1204B (or, for example, toward road marking 1204C), host vehicle 200 can assume that target vehicle 802 is leaving road 1202, and host vehicle 802 can accelerate or continue to maintain its speed.

[0189] In some embodiments, the navigation action may also depend on a predicted trajectory of the target vehicle. As an example, referring to Fig. 12A, if the predicted trajectory indicates that the target vehicle 802 will exit the road 1202, the host vehicle 200 may accelerate to reduce the distance between the host vehicle 200 and the target vehicle 800 and / or pass the target vehicle 802 on the left. As another example, if the predicted trajectory indicates that the target vehicle 802 will remain on the road 1202, the host vehicle 200 may decelerate to maintain a safe distance between the host vehicle 200 and the target vehicle 802.

[0190] In some embodiments, the particular navigation action for the host vehicle may include lateral control of the host vehicle. For example, as in Fig. 12A, the road 1202 splits to form an exit lane, and the host vehicle 200 may remain on the road 1202 to reach its intended destination rather than using the exit lane to exit the road 1202. Accordingly, the processing unit 110 may identify lane marker 1204A (e.g., via analysis of one or more images captured by one or more image capture devices) as a "host left" lane marker delineating a left edge of the "host lane" and determine that the host vehicle should maintain lateral control while negotiating the exit lane. In particular, the navigation action determined by the processing unit 110 may include maintaining the host vehicle 200 at a constant lateral distance from the lane marker 1204A to remain on the road 1202.Alternatively, when the host vehicle 200 uses the exit lane to exit roadway 1202 to reach its intended destination, the processing unit 110 may identify lane marker 1204D (e.g., by analyzing one or more images captured by one or more image capture devices) as a "near right" lane marker delineating the right edge of the exit lane and determine that the host vehicle should maintain lateral control while exiting roadway 1202. Specifically, the navigation action determined by the processing unit 110 may include maintaining the host vehicle 200 at a constant lateral distance from lane marker 1204D to exit roadway 1202.

[0191] Alternatively or additionally, the particular navigation action for the host vehicle may include steering the host vehicle along a curvature of the road. For example, if, as in Fig. 12A, splits road 1202 to form an exit lane, and host vehicle 200 is to remain on road 1202, processing unit 110 may identify lane marker 1204A, which is a "host left" lane marker marking a left edge of the "host lane," to estimate a curvature of road 1202, while ignoring the use of lane marker 1204D, which is a "near right" lane marker marking a right edge of the exit lane.

[0192] In some embodiments, in addition to determining a navigation action for the host vehicle, or as part of determining a navigation action for the host vehicle, processing unit 110 may predict a navigation action of the target vehicle. For example, processing unit 110 may predict the navigation action of the target vehicle based on the type of lane in which the target vehicle is traveling. If the target vehicle is traveling in a split lane, processing unit 110 may predict that the target vehicle will exit the roadway.

[0193] For example, related to Fig. 12A, the processing unit 110 may identify a position of the target vehicle 802 relative to the lane marker 1204C, which is a split lane marker, to determine whether the target vehicle 802 will exit the road 1202. For example, if the lateral distance between the target vehicle 802 and the lane marker 1204C is constant or decreasing, the processing unit 110 may determine that the target vehicle 802 is exiting the road 1202. Conversely, if the processing unit 110 determines that the lateral distance between the target vehicle 802 and the lane marker 1204C is increasing, the processing unit 110 may determine that the target vehicle 802 will not exit the road 1202.In this way, processing unit 110 can determine whether target vehicle 802 is following the exit lane and, accordingly, whether target vehicle 802 is exiting roadway 1202 before target vehicle 802 crosses lane marker 1204B into the exit lane. In some embodiments, the navigation action of the target vehicle can be determined based on the predicted trajectory. For example, if target vehicle 802 is predicted to exit roadway 1202, host vehicle 200 can accelerate or maintain its speed.

[0194] In some embodiments, in addition to identifying the lane markers 1204A-1204D, the processing unit 110 may perform an identification of the lane markers 1204A-1204D. The determined lane marker(s) may determine a type of lane in which the target vehicle and the host vehicle are traveling. In some embodiments, the determined lane marker identification(s) may determine the type of lane as a lane merge lane, such as a "lane merge," a "lane merge to," or a "lane merge from," as described above in connection with Fig. 8A. Alternatively, in some embodiments, the detected indicia(s) of the lane marking(s) may determine the type of lane as a split lane, such as a “split” split lane, an “open” split lane, or an “extended” split lane, as described above in connection with Fig. 8B. The generation of the labels may include feeding the image(s) to a neural network as described above in connection with Fig. 9. In addition, the marked image may have any of the features described above in connection with Fig. 9 described forms.

[0195] Example markings for lane markings are shown in Fig. 12B. As shown, the processing unit 110 may identify and label markers in a first image 1206. For example, the lane marker 1204A may be labeled as a "host left" lane marker (labeled "HL"), indicating the left side of a "host" lane in which the host vehicle 200 is traveling. The lane marker 1204B may be labeled as a "host right" lane marker (labeled "HR"), indicating the right side of a "host" lane in which the host vehicle 200 is traveling.

[0196] As in Fig. 12B, the processing unit 110 may additionally identify and label lane markers in a second image 1208. As shown, lane markers 1204A, 1204B, and 1204D may be labeled in the second image 1208. The processing unit 110 may relabel lane marker 1204A with a "host left" lane marker and lane marker 1204B with a "host right" lane marker. Lane marker 1204D, which was not visible in the first image 1206, may be labeled in the second image 1208 as a "right next" lane marker, indicating the right side of the lane to the right of the lane in which the host vehicle 200 is traveling.

[0197] Although Fig. 12B shows a target vehicle 802, it should be understood that in some embodiments, the labels of the lane markers may be used to determine a navigation action for the host vehicle 200 even when no target vehicle is present. For example, in some embodiments, a neural network fed with labeled images as described above may be configured to detect a lane separation based on labeled lane markers, as shown in Fig. 12B, and the host vehicle 200 may take a navigation action to account for the lane separation.

[0198] As described above, the detected lane marking may determine a type of lane in which the target vehicle and the host vehicle are traveling. Accordingly, the processing unit 110 may determine the type of lane in which the target vehicle is traveling based on the identified lane markings and / or labels.

[0199] In some embodiments, determining the type of lane in which the target vehicle is traveling may include determining a change in the lateral distance between at least two of the marked lane markers. For example, if the lateral distance between the at least two marked lane markers increases, the processing unit 110 may determine that the lane in which the target vehicle is traveling is a split lane. For example, with reference to Fig. 12B, the at least two identified lane markers may be lane markers 1204B and 1204D. Processing unit 110 may determine a change in the lateral distance between lane markers 1204B and 1204D to determine the type of lane in which target vehicle 802 is traveling. As shown, the lateral distance between lane markers 1204B and 1204D increases. That is, at a closer distance to host vehicle 200, the lateral distance between lane markers 1204B and 1204D is less than the lateral distance between lane markers 1204B and 1204D at a farther distance from host vehicle 200. Accordingly, processing unit 110 may determine that target vehicle 802 is traveling in a split lane. In some embodiments, processing unit 110 may verify a particular type of lane using map data (e.g., from map database 160).

[0200] Alternatively or additionally, in some embodiments, determining the type of lane in which the target vehicle is traveling may include determining a change in the lane marker indicia between images or between lane markers in a particular image. Such a determination may allow processing unit 110 to further determine what type of lane merge (e.g., "merge," "merge to," or "merge from") or what type of lane split (e.g., "lane split," "lane split open," or "lane widening") is occurring on road 1202. For example, with respect to Fig. 12B, the processing unit 110 may determine that the first image 12066 includes a "host left" lane marker (labeled "HL") and a "host right" lane marker (labeled "HR"), as described above. The processing unit 110 may further determine that a second image 1208 still includes a "host left" lane marker (labeled "HL") and a "host right" lane marker (labeled "HR"), but also includes a "next right" lane marker (labeled "NRR"). Based on these lane markers, the processing unit 110 may determine that the "next right" lane, indicated in part by the "next right lane" lane marker in the second image 1206, extends from the "host" lane in which the host vehicle 200 is traveling. Accordingly, the processing unit 110 may determine that the paved vehicle 802 is traveling in a split lane with “lane widening.”Because the “lane extension” is located on the right side of road 1202, processing unit 110 may determine that the “next right” lane in which target vehicle 802 is traveling is an exit lane, meaning that host vehicle 200 may prepare for target vehicle 802 to exit road 1202.

[0201] Fig. 13 is a flowchart illustrating another exemplary process 1300 for determining one or more navigation actions based on a lane change analysis in accordance with the disclosed embodiments. The process 1300 may be performed, for example, by the processing unit 110 described above. For illustrative purposes, the process 1300 is described with reference to Fig. 14A-14C, which are illustrations of an exemplary lane change analysis for lane merging in accordance with the disclosed embodiments, and with reference to Fig. 15A-15C, which are illustrations of an exemplary lane change analysis for a lane separation in accordance with the disclosed embodiments. In the following discussion, the autonomous vehicle is referred to as the host vehicle (e.g., vehicle 200) and may include one or more components of system 100, as described above in connection with Fig. 1 described.

[0202] As illustrated, process 1300 begins in step 1302 by receiving a plurality of images from an image capture device representing an environment of the host vehicle. The environment includes a road on which the host vehicle is traveling. The image capture device 122 and the images may take any of the forms described above for the image capture device 122 and the images in connection with Fig. 19 were described.

[0203] An example image 1400 captured by an image capture device 122 and depicting a road 1402 with a lane merge is shown in Fig. 14A. As shown in Fig. 14A, the image 1400 may depict an environment of the host vehicle 200. For example, the image 1400 may include depictions of a road 1402 on which the host vehicle 200 is traveling and the lane markings 1404A-1404E on the road 1402. The road 1402 and the lane markings 1404A-1404E may take any of the forms described above for the road 1002 and the lane markings 1004A-1004D in connection with Fig. 10 were described.

[0204] Another example image 1500, which was taken by an image recording device 122 and represents a road 1502 with a lane separation, is shown in Fig. 15A. As shown in Fig. 15A, the image 1500 may depict an environment of the host vehicle 200. The image 1500 may include, for example, depictions of a road 1502 on which the host vehicle 200 is traveling and lane markings 1504A-1504E. The road 1502 and lane markings 1504A-1504E may take any of the forms described above for the road 1002 and lane markings 1004A-1004D in connection with Fig. 10 were described.

[0205] In order to Fig. 13, the processing unit 110 may analyze at least one of the plurality of images in step 1304 to identify a target vehicle. As previously mentioned, the target vehicle may be in the same lane as the host vehicle or in a different lane than the host vehicle. For example, as shown in Fig. 14A, in some embodiments, the target vehicle 802 may travel in front of the host vehicle 200 in a different lane than the lane in which the host vehicle 200 is traveling as the host vehicle 200 and the target vehicle 802 approach the lane merge. As another example, as shown in Fig. 15A, in some embodiments, the target vehicle 802 may be traveling in front of the host vehicle 200 in the same lane as the host vehicle 200 as the host vehicle 200 and the target vehicle 802 approach the lane separation. The processing unit 110 may analyze one or more of the captured images to identify the target vehicle 802, for example, using the monocular image analysis module 402 and / or the stereo image analysis module 404 described above in connection with Fig. 4 were described.

[0206] In step 1306, the processing unit 110 may analyze at least one of the plurality of images to identify a plurality of lane markers associated with the lane in which the target vehicle is traveling. For example, as shown in Fig. 14A, analyze the image 1400 to identify one or more lane markers 1404A-1404E. The processing unit 110 may analyze the image 1400 to identify the lane markers 1404A-1404E, for example, using the monocular image analysis module 402 and / or the stereo image analysis module 404 described above in connection with Fig. 4, and / or in any other manner described above. As a further example, as in Fig. 15A, the processing unit may analyze the image 1500 to identify one or more lane markers 1504A-1504E. The processing unit 110 may analyze the image 1500 to identify the lane markers 1504A-1504E, for example, using the monocular image analysis module 402 and / or the stereo image analysis module 404 described above in connection with Fig. 4 and / or in any other manner described above.

[0207] In order to Fig. 13, the processing unit 110 may determine a lane marker label for at least one of the identified lane markers in step 1308. In some embodiments, determining the labels may involve feeding the image(s) to a neural network, such as those described above in connection with Fig. 4 and / or a convolutional neural network. The neural network may be trained using labels and / or lane marker identifiers associated with one or more images. Labeled data or images associated with labels and / or lane marker identifiers may be received by the neural network from a variety of resources, e.g., one or more databases. The one or more databases may be located locally (e.g., included within system 100) and / or remotely (e.g., via a network such as a cellular network and / or the Internet, etc.) and may be received, e.g., via a wireless transceiver 172 of system 100. Labeling of the stored images is performed, e.g., by storing a label (e.g., a lane merge or lane split) associated with an identified lane marker.In some embodiments, not every identified lane marker may be labeled. Rather, the labeling may be limited to those lane markers relevant for detecting a lane change and determining a lane type, as described below. Finally, the neural network could be capable of automatically identifying and labeling the lane markers.

[0208] An example of a labeled image 1406 that can be used, for example, by a neural network is shown in Fig. 14B. As shown, the labeled image 1406 includes a representation of the road 1402 with the lane merge, and the lane markers 1404A-1404E have been labeled, as shown by the different line patterns. For example, the dashed lines on lane markers 1404A, 1404B, and 1404E may indicate "normal" lane markers that are not part of the lane merge, the small dashed line on lane marker 1404C may mark the inner portion of a lane merge, and the solid line on lane marker 1404D may indicate the outer portion of a lane merge. Another example of labeling of image 1506 that may be used by a neural network is shown in Fig. 15B. As shown, the annotated image 1506 includes a representation of the road 1502 with the lane separation, and the lane markings 1504A-1504E have been annotated, as shown by the different line patterns. For example, the dashed lines on lane markings 1504A, 1504B, and 1504C may mark "normal" lane markings that are not part of the lane separation, the small dashed line on lane marking 1504D may mark the inner portion of a divided lane, and the solid line on lane marking 1504E may mark the outer portion of a divided lane. While Fig. 14B and Fig. 15B depict the markings in the form of different line patterns, in some embodiments, the markings may alternatively or additionally take the form of coloring, alphabetic and / or numeric marking, or other indicators associated with the marked lane markings. The neural network may, for example, use the marked image 1406 and / or 1506 to enhance the lane marking through machine learning.

[0209] In some embodiments, lane markers may be provided with a label indicating the relationship of the lane marker to the host vehicle 200. For example, in Fig. 14C and the annotated image 1410, the lane marker 1404B may be labeled "Host Left" ("HL", as shown), indicating the left side of a "Host" lane in which the host vehicle 200 is traveling, and the lane marker 1404C may be labeled "Host Right" ("HR", as shown), indicating the right side of the "Host Lane." Additionally, the lane marker 1404A may be labeled "Next Left Lane" ("NLL", as shown), indicating the left side of a "Next Left Lane" immediately left of the "Host Lane." The lane marker 1404B, in addition to a "Host Left" lane marker, may be labeled "Next Left Lane Right" ("NLR", as shown), indicating the right side of the "Next Left" lane immediately left of the "Host" lane.Lane marker 1404E may be labeled as a "next right lane" ("NRR," as shown), indicating the right side of the "next right lane" immediately to the right of the "host" lane. Lane markers 1404C and 1404D, which form the lane merge, may be labeled as "inner" and "outer" lane markers, as shown: Lane marker 1404C may be labeled as an "inner lane marker" ("ILM," as shown), and lane marker 1404D may be labeled as an "outer lane marker" ("OLM," as shown).

[0210] As another example, referring to Fig. 15C and annotated image 1510, lane marker 1504B may be labeled as a "Host Left" lane marker ("HL", as shown), indicating the left side of a "Host" lane in which host vehicle 200 is traveling, and lane marker 1504C may be labeled as a "Host Right" lane marker ("HR", as shown), indicating the right side of the "Host" lane. Additionally, lane marker 1504A, as shown, may be labeled as a "Next Left Lane" lane marker ("NLL", as shown), indicating the left side of a "Next Left Lane" immediately to the left of the "Host" lane. The lane marker 1504B, in addition to a "Host Left" lane marker, may be marked as a "Next Left Lane Right" ("NLR" as shown) lane marker, indicating the right side of the "Next Left" lane immediately to the left of the "Host" lane.Lane markings 1504D and 1504E, which form the lane separation as shown, may be identified as "inner" and "outer" lane markings: lane marking 1504D may be identified as an "inner lane marking" ("ILS", as shown) and lane marking 1504E as an "outer lane marking" ("OLS", as shown).

[0211] In some embodiments, the lane marker identifiers may be stored in a database (e.g., a relational database). For example, lane marker identifiers (e.g., identifiers corresponding to lane marker labels and / or lane marker types) may be stored in the database in association with lane marker identifiers and / or lane marker locations (e.g., geographic coordinates). In some embodiments, the database containing the lane marker identifiers and / or lane marker locations may be included, for example, in a memory (e.g., memory 150) of the system 100 and / or in a memory of a remote server accessible by the processing unit 110 of the system 100 over a network.

[0212] In order to Fig. 13, the processing unit 110 may determine a predicted trajectory of the target vehicle in step 1310. For example, the processing unit 110 may determine a predicted trajectory of the target vehicle based on analyzing data from one or more sensors and / or devices of the host vehicle (e.g., an image sensor, lidar, radar, a GPS device, a speed sensor, an accelerometer, a suspension sensor, an acoustic sensor, etc.).

[0213] In some embodiments, processing unit 110 may determine the predicted trajectory of the target vehicle based on the analysis of at least two of the plurality of images captured by at least one image capture device of the host vehicle. The analysis of the images may, for example, take one of the forms described above, for example, in step 1304. In some processes, processing unit 110 may determine the predicted trajectory of the target vehicle based, for example, on the lane marking(s) of the lane in which the target vehicle is traveling. Referring to Fig. 14C, for example, the processing unit 110 may determine that markers 1404C and 1404D are labeled "inner lane merge" and "outer lane merge," respectively, indicating that the target vehicle 802 is traveling in a lane that merges with the lane in which the host vehicle 200 is traveling in front of the host vehicle 200. Accordingly, the processing unit 110 may determine that the predicted trajectory of the target vehicle 802 merges into the lane in which the host vehicle 200 is located. In some embodiments, by predicting the trajectory of the target vehicle 802 using lane markers, the processing unit 110 may determine whether the target vehicle 802 will merge into the lane in which the host vehicle 200 is traveling even before the target vehicle 802 crosses lane marker 1404C into the lane. As another example, related to Fig. 15C, the processing unit 110 may determine that markers 1504D and 1504E are labeled "inside lane split" and "outside lane split," respectively, indicating that the target vehicle 802 is traveling in a lane that splits in front of the host vehicle 200 to form an exit lane. Accordingly, the processing unit 110 may determine that the predicted trajectory of the target vehicle 802 is to exit the lane in which the host vehicle 200 is traveling and enter the exit lane. In some embodiments, by predicting the trajectory of the target vehicle 802 using lane markers, the processing unit 110 may determine whether the target vehicle 802 is exiting the exit lane even before the target vehicle 802 crosses the lane marker 1504D in the exit lane.

[0214] Alternatively or additionally, in order to determine a predicted trajectory of the target vehicle 802, the processing unit 110 may position the target vehicle 802 relative to one or more lane markers. With reference to Fig. 14C, for example, the processing unit 110 may determine that the target vehicle 802 is merging into the lane in which the host vehicle 200 is located when the lateral distance between the target vehicle 802 and the lane marker 1404E is constant or increasing. In this way, it may be determined whether the target vehicle 802 is tracking the lane merge and, accordingly, whether the target vehicle 802 is entering the lane in which the host vehicle 200 is traveling, even before the target vehicle 802 crosses the lane marker 1404C on the path.

[0215] As another example, related to Fig. 15C, the processing unit 110 may determine that the target vehicle 802 is exiting the road 1502 by splitting into the newly formed road marking when the lateral distance between the target vehicle 802 and the lane marking 1504E is constant or decreasing. In this way, it may be determined whether the target vehicle 802 is following the exit lane and whether the target vehicle 802 is exiting the road 1502 even before the target vehicle 802 crosses the lane marking 1504D in the exit lane.

[0216] In order to Fig. 13, the processing unit 110 may determine a navigation action for the host vehicle in step 1312. In some embodiments, the processing unit 110 may determine the navigation action for the host vehicle based on the determined lane marker label or the predicted trajectory of the target vehicle. In some embodiments, the processing unit 110 may determine the navigation action for the host vehicle based on the determined lane marker label and the predicted trajectory of the target vehicle.

[0217] For example, related to Fig. 14C, if the lane marking indicia 1404C and 1404D indicate that the target vehicle 802 is traveling in a merge lane and / or the predicted trajectory indicates that the target vehicle 802 is merging into the lane in which the host vehicle 200 is traveling, the host vehicle 200 may decelerate to maintain a safe distance between the host vehicle 200 and the target vehicle 802. As another example, referring to Fig. 15C, the host vehicle 200 may accelerate to reduce the distance between the host vehicle 200 and the target vehicle 802 and / or to pass the target vehicle 802 on the left when the lane markers 1504D and 1504E indicate that the target vehicle 802 is traveling in the exit lane and / or the predicted trajectory indicates that the target vehicle 802 will exit the roadway 1502.

[0218] The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments. Although aspects of the disclosed embodiments are described as being stored in memory, one skilled in the art will recognize that these aspects may also be stored on other types of computer-readable media, such as secondary storage, e.g., hard drives or CD-ROM, or other forms of RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray, or other optical drives.

[0219] Computer programs based on the written description and the disclosed methods are easy to use for a skilled developer. The various programs or program modules can be created using techniques known in the art or designed in conjunction with existing software. For example, program sections or program modules can be designed in or using the .NET Framework, .NET Compact Framework (and related languages ​​such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML / AJAX combinations, XML, or HTML with included Java applets.

[0220] Although illustrative embodiments have been described herein, the scope of the claimed invention is indicated by the following claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] EP-A1-2 455 266

[0000] US-A1-2005 015 203

[0000] US-A1-2014 236 414

[0000] US 2014 / 0236414 A1

[0005] US 2005 / 0015203 A1

[0005] US 7,151,996

[0146] Cited non-patent literature

[0000] ARMINGOL ET AL: “IVVI: Intelligent vehicle based on visual information”, ROBOTICS AND AUTONOMOUS SYSTEMS, ELSEVIER SCIENCE PUBLISHERS, AMSTERDAM, NL, vol. 55, no. 12, 5 November 2007 (2007-11-05), pages 904-916, XP022328739, ISSN: 0921-8890, DOI: 10.1016 / J.ROBOT.2007.09.

[0001] Armingol, Jose Maria, et al, „IVVI: Intelligent vehicle based on visual information“, Robotics and Autonomous Systems, Elsevier Science Publishers, Amsterdam, NL, Bd. 55, Nr. 12, 5. November 2007, Seiten 904-916

[0005]

Claims

[1] An apparatus comprising a processor of a host vehicle (200), the processor being configured to cause the host vehicle to: Receiving (902), from an image capture device (122), a plurality of images representative of an environment (1000) of the host vehicle (200), the environment comprising a road on which the host vehicle is traveling; analyzing (904) at least one of the plurality of images to identify a target vehicle (802) traveling in a lane of the road that is different from a lane in which the host vehicle (200) is traveling; analyzing (906) at least one of the plurality of images to identify at least one lane marker (1004A, 1004B) associated with the lane in which the target vehicle is traveling; Determining (912) at least one feature of the target vehicle (800) based on the plurality of images, wherein the at least one determined feature of the target vehicle (800) comprises a detected lateral movement of the target vehicle with respect to the at least one identified lane marker (1004A, 1004B); and Determining (914) a navigation action for the host vehicle (200) based on the determined feature of the target vehicle (800). [2] The apparatus of claim 1, wherein the processor is configured to cause the host vehicle to perform the steps of claim 1, further comprising detecting a distance of the at least one identified lane marker to a reference point. [3] The apparatus of claim 2, wherein the processor is configured to cause the host vehicle to perform the steps of claim 2, wherein the reference point comprises at least one further lane marking of the road. [4] The apparatus of any one of claims 1 to 3, wherein the processor is configured to cause the host vehicle to perform the steps of any one of claims 1 to 3, further comprising detecting an intersection of the at least one identified lane marking with at least one other lane marking of the road. [5] The apparatus of any one of claims 1 to 4, wherein the processor is configured to cause the host vehicle to perform the steps of any one of claims 1 to 4, further comprising detecting a change in lateral distances between the at least one identified lane marker and at least one other lane marker of the road. [6] The apparatus of any one of claims 1 to 5, wherein the processor is configured to cause the host vehicle to perform the steps of any one of claims 1 to 5, wherein determining at least one feature of the target vehicle comprises analyzing data from one or more sensors of the host vehicle. [7] The apparatus of any one of claims 1 to 6, wherein the processor is configured to cause the host vehicle to perform the steps of any one of claims 1 to 6, wherein the at least one particular feature of the target vehicle comprises one or more of a position of the target vehicle on the road ahead of the host vehicle (200), a speed of the target vehicle relative to the host vehicle (200), and / or a predicted trajectory of the target vehicle. [8] The apparatus of any one of claims 1 to 7, wherein the processor is configured to cause the host vehicle to perform the steps of any one of claims 1 to 7, wherein the determined navigation action for the host vehicle (200) comprises changing an acceleration rate. [9] The apparatus of any one of claims 1 to 8, wherein the processor is configured to cause the host vehicle to perform the steps of any one of claims 1 to 8, wherein the particular navigation action for the host vehicle (200) comprises a steering action of the host vehicle (200). [10] The apparatus of any one of claims 1 to 9, wherein the processor is configured to cause the host vehicle to perform the steps of any one of claims 1 to 9, wherein the at least one identified lane marker comprises a merge lane or a split lane. [11] The apparatus of any one of claims 1 to 10, wherein the processor is configured to cause the host vehicle to perform the steps of any one of claims 1 to 10, including detecting a separation of the at least identified lane marking into another lane marking. [12] The apparatus of any one of claims 1 to 9, wherein the processor is configured to cause the host vehicle to perform the steps of any one of claims 1 to 9, further comprising: detecting (908) one or more lane marking features of the at least one identified lane marking; Determining (910) a type of the at least one identified lane marker using the one or more detected lane marker features; and predicting a navigation action of the target vehicle based on the determined type of the at least one lane marker. [13] The apparatus of claim 10, wherein the processor is configured to cause the host vehicle to perform the steps of claim 10, wherein the predicted navigation action of the target vehicle comprises lane merging. [14] A computer-readable storage medium storing instructions that, when executed by at least one processor of a system for navigating a host vehicle (200), cause the system to perform the steps of any one of claims 1 to 13. [15] A system for navigating a host vehicle, the system comprising an apparatus according to any one of claims 1 to 13.

Images