Vehicle navigation system, vehicle, and program
The system in autonomous vehicles improves lane change prediction and response by analyzing images and sensor data, reducing collisions and enhancing traffic efficiency through adjusted navigation strategies.
Patent Information
- Application Number
- JP2022129417
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2016-07-12
- Filing Date
- 2022-08-15
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2036-11-23
AI Technical Summary
Autonomous vehicles face challenges in predicting whether other vehicles will attempt to cut into their lane, leading to potential collisions or uncomfortable rides due to delayed navigation responses, and need to balance between necessary and optional interruptions while considering altruistic behavior.
The system uses cameras and processing units to analyze images, combined with GPS and sensor data, to detect potential lane changes by other vehicles and adjust navigation responses based on sensitivity modifiers and altruistic behavior parameters.
Enhances the prediction of lane change attempts, reducing unnecessary braking and collisions while allowing for efficient traffic flow by considering both safety and altruistic behavior.
Smart Images

Figure 0007582625000001 
Figure 0007582625000002 
Figure 0007582625000003
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 260,281, filed November 26, 2015, and U.S. Provisional Patent Application No. 62 / 361,343, filed July 12, 2016, all of which are incorporated herein by reference in their entirety.
[0002] background Technical Field The present disclosure relates generally to autonomous vehicle navigation. Additionally, the present disclosure relates to systems and methods for detecting and responding to cutting-in vehicles and navigating while taking into account altruistic operating parameters. [Background technology]
[0003] Background information As technology continues to evolve, the goal of fully autonomous vehicles capable of navigating the roads becomes more realistic. An autonomous vehicle may need to consider various factors and, based on those factors, may make appropriate decisions to safely and accurately reach an intended destination. For example, an autonomous vehicle may need to process and interpret visual information (e.g., information captured from a camera) and may also use information obtained from other sources (e.g., from a Global Positioning System (GPS) device, speed sensors, accelerometers, suspension sensors, etc.). At the same time, to navigate to a destination, an autonomous vehicle may need to identify its location within a particular road (e.g., a particular lane within a multi-lane road), navigate alongside other vehicles, avoid obstacles and pedestrians, observe traffic signals and signs, and drive from one road to another at appropriate intersections or interchanges.
[0004] During navigation, an autonomous vehicle may encounter another vehicle attempting to lane shift. For example, a vehicle in a lane to the left or right of the lane in which the autonomous vehicle is traveling may attempt to change, i.e., cut in, into the lane in which the autonomous vehicle is traveling. When such a cut in occurs, the autonomous vehicle must make a navigational response, for example, by changing its speed or acceleration and / or changing into another lane, to avoid being cut in by the other vehicle. Summary of the Invention [Problem to be solved by the invention]
[0005] In some instances, the other vehicle may appear to be attempting to cut in but ultimately does not do so (e.g., because the driver of the other vehicle changes his or her mind or the other vehicle is simply skidding). Although delaying execution of a navigation response until there is a reasonable likelihood that the other vehicle will cut in may prevent unnecessary braking, such a delay may also increase the risk of a collision and / or cause braking that may be uncomfortable for occupants in the autonomous vehicle. Thus, there is a need for improved prediction of cut-in attempts by vehicles.
[0006] Furthermore, in some cases, cutting in by other vehicles may be required, for example, by road and / or traffic rules. However, in other cases, cutting in may be optional, for example, when other vehicles simply want to pass a slower vehicle. Because an autonomous vehicle may be programmed to safely drive to a destination on time, the autonomous vehicle may not necessarily allow other vehicles to cut in if the cutting in is not necessary. However, in some cases, it may be preferable for the operator of the autonomous vehicle and / or for the overall efficiency of traffic to allow such cutting in. Thus, a cutting in process that includes altruistic behavior is necessary. [Means for solving the problem]
[0007] overview Embodiments according to the present disclosure provide systems and methods for autonomous vehicle navigation. The disclosed embodiments may use cameras to provide autonomous vehicle navigation features. For example, according to embodiments of the present disclosure, the disclosed system may include one, two, or more cameras that monitor the vehicle's environment. The disclosed system may provide a navigation response, for example, based on an analysis of images captured by one or more of the cameras. The navigation response may also take into account other data, including, for example, global positioning (GPS) data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data.
[0008] According to a disclosed embodiment, a vehicle interruption detection and response system for a host vehicle is provided, which may include a data interface and at least one processing device. The at least one processing device may be programmed to: receive a plurality of images from at least one image capture device associated with the host vehicle via the data interface; identify in the plurality of images an image of a target vehicle traveling in a first lane that is different from a second lane in which the host vehicle is traveling; identify at least one indicator that the target vehicle will change from the first lane to the second lane based on an analysis of the plurality of images; detect whether at least one predetermined interruption sensitivity modifier is present in an environment of the host vehicle; if the predetermined interruption sensitivity modifier is not detected, cause a first navigation response in the host vehicle based on the identification of the at least one indicator and based on a value associated with a first interruption sensitivity parameter; and if the at least one predetermined interruption sensitivity modifier is detected, cause a second navigation response in the host vehicle based on the identification of the at least one indicator and based on a value associated with a second interruption sensitivity parameter, the second interruption sensitivity parameter being different from the first interruption sensitivity parameter.
[0009] According to other disclosed embodiments, the host vehicle may include a vehicle body, at least one image capture device, and at least one processing device. The at least one processing device may be programmed to: receive a plurality of images from at least one image capture device associated with the host vehicle; identify in the plurality of images an image of a target vehicle traveling in a first lane that is different from a second lane in which the host vehicle is traveling; identify at least one indicator that the target vehicle will change from the first lane to the second lane based on an analysis of the plurality of images; detect whether at least one predetermined interruption sensitivity modifier is present in an environment of the host vehicle; and if the predetermined interruption sensitivity modifier is not detected, generate a first navigation response in the host vehicle based on the identification of the at least one indicator and based on a value associated with a first interruption sensitivity parameter; and if the at least one predetermined interruption sensitivity modifier is detected, generate a second navigation response in the host vehicle based on the identification of the at least one indicator and based on a value associated with a second interruption sensitivity parameter, the second interruption sensitivity parameter being different from the first interruption sensitivity parameter.
[0010] According to yet another disclosed embodiment, a method of detecting and responding to a cut-in by a target vehicle is provided, the method including receiving a plurality of images from at least one image capture device associated with a host vehicle, identifying in the plurality of images an image of the target vehicle traveling in a first lane different from a second lane in which the host vehicle is traveling, identifying at least one indicator that the target vehicle will change from the first lane to a second lane based on an analysis of the plurality of images, detecting whether at least one predefined cut-in sensitivity modifier is present in an environment of the host vehicle, causing a first navigation response in the host vehicle based on the identification of the at least one indicator and based on a value associated with a first cut-in sensitivity parameter if the predefined cut-in sensitivity modifier is not detected, and causing a second navigation response in the host vehicle based on the identification of the at least one indicator and based on a value associated with a second cut-in sensitivity parameter if the at least one predefined cut-in sensitivity modifier is detected, the second cut-in sensitivity parameter being different from the first cut-in sensitivity parameter.
[0011] According to a disclosed embodiment, a navigation system for a host vehicle is provided. The system may include a data interface and at least one processing device. The at least one processing device may be programmed to receive a plurality of images from at least one image capture device associated with the host vehicle via the data interface, identify at least one target vehicle in the environment of the host vehicle based on an analysis of the plurality of images, identify one or more situational features associated with the target vehicle based on an analysis of the plurality of images, identify a current value associated with an altruistic behavior parameter, and determine that a change in a navigation state of the host vehicle is not required based on the one or more situational features associated with the target vehicle, but cause at least one navigation change in the host vehicle based on the current value associated with the altruistic behavior parameter and based on the one or more situational features associated with the target vehicle.
[0012] According to another disclosed embodiment, the host vehicle may include a body, at least one image capture device, and at least one processing device. The at least one processing device may be configured to receive a plurality of images from the at least one image capture device, identify at least one target vehicle in an environment of the host vehicle based on an analysis of the plurality of images, identify one or more situational features associated with the target vehicle based on an analysis of the plurality of images, identify a current value associated with an altruistic behavior parameter, and determine that a change in a navigation state of the host vehicle is not required based on the one or more situational features associated with the target vehicle, but cause at least one navigation change in the host vehicle based on the current value associated with the altruistic behavior parameter and based on the one or more situational features associated with the target vehicle.
[0013] According to yet another disclosed embodiment, a method of navigating a host vehicle is provided, which may include receiving a plurality of images from at least one image capture device associated with the vehicle, identifying at least one target vehicle in an environment of the host vehicle based on an analysis of the plurality of images, determining one or more situational features associated with the target vehicle based on an analysis of the plurality of images, determining a current value associated with an altruistic behavior parameter, and determining that a change in a navigation state of the host vehicle is not required based on the one or more situational features associated with the target vehicle, but causing at least one navigational change in the host vehicle based on the current value associated with the altruistic behavior parameter and based on the one or more situational features associated with the target vehicle.
[0014] According to other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that are executed by at least one processing device and that perform any of the methods described herein.
[0015] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the scope of the claims.
[0016] BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. [Brief description of the drawings]
[0017] [Figure 1] 1 is a diagrammatic representation of an exemplary system according to a disclosed embodiment. [Figure 2A] 1 is a side view representation of an exemplary vehicle including a system according to disclosed embodiments. [Figure 2B] 2B is a top view representation of the vehicle and system shown in FIG. 2A according to a disclosed embodiment. [Figure 2C]1 is a top view representation of another embodiment of a vehicle including a system according to the disclosed embodiments. [Figure 2D] 1 is a top view representation of yet another embodiment of a vehicle including a system according to disclosed embodiments. [Figure 2E] 1 is a top view representation of yet another embodiment of a vehicle including a system according to disclosed embodiments. [Figure 2F] 1 is a diagrammatic representation of an exemplary vehicle control system in accordance with a disclosed embodiment. [Figure 3A] 1 is a diagrammatic representation of the interior of a vehicle including a rearview mirror and a user interface of a vehicle imaging system according to disclosed embodiments. [Figure 3B] 1 is a diagram of an example camera mount configured to be positioned behind a rearview mirror, facing a vehicle windshield, in accordance with a disclosed embodiment. [Figure 3C] 3C is a diagram of the camera mount shown in FIG. 3B from a different perspective, according to a disclosed embodiment. [Figure 3D] 1 is a diagram of an example camera mount configured to be positioned behind a rearview mirror, facing a vehicle windshield, in accordance with a disclosed embodiment. [Figure 4] FIG. 1 illustrates an example block diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments. [Figure 5A] 1 is a flowchart illustrating an example process for generating one or more navigational responses based on monocular image analysis, according to disclosed embodiments. [Figure 5B] 1 is a flowchart illustrating an example process for detecting one or more vehicles and / or pedestrians in a set of images, according to disclosed embodiments. [Figure 5C] 1 is a flowchart illustrating an example process for detecting road markings and / or lane geometry information in a set of images, according to disclosed embodiments. [Figure 5D] 1 is a flowchart illustrating an example process for detecting traffic lights in a set of images, according to disclosed embodiments. [Figure 5E] 1 is a flowchart of an example process for generating one or more navigational responses based on a vehicle path, according to disclosed embodiments. [Figure 5F] 4 is a flowchart illustrating an example process for determining whether a leading vehicle is changing lanes, according to disclosed embodiments. [Figure 6] 1 is a flowchart illustrating an example process for generating one or more navigational responses based on stereo image analysis, according to disclosed embodiments. [Figure 7] 1 is a flowchart illustrating an example process for generating one or more navigational responses based on an analysis of three sets of images, according to disclosed embodiments. [Figure 8] FIG. 1 illustrates an example block diagram of a memory configured to store instructions to perform one or more operations in accordance with the disclosed embodiments. [Figure 9A] 1 is a diagram of an example situation in which a vehicle may detect and respond to an interruption, according to the disclosed embodiments. [Figure 9B] 1 illustrates an exemplary predefined interrupt sensitivity modifier according to the disclosed embodiments. [Figure 9C] 1 illustrates an exemplary predefined interrupt sensitivity modifier according to the disclosed embodiments. [Figure 9D] 1 illustrates an exemplary predefined interrupt sensitivity modifier according to the disclosed embodiments. [Figure 9E] 1 illustrates an exemplary predefined interrupt sensitivity modifier according to the disclosed embodiments. [Figure 10] 1 is a diagram of an example situation in which a vehicle may engage in altruistic behavior, according to disclosed embodiments. [Figure 11] 4 is a flowchart illustrating an example process for vehicle interruption detection and response, according to disclosed embodiments. [Figure 12] 12 is a flowchart illustrating an example process 1200 for navigating while taking into account altruistic behavior, according to a disclosed embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] Detailed Description The following detailed description refers to the accompanying drawings. Whenever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. Although several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components shown in the drawings, and the exemplary methods described herein may be modified by substituting, reordering, deleting, or adding steps of the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the appropriate scope is defined by the appended claims.
[0019] Autonomous Vehicles Overview As used throughout this disclosure, the term "autonomous vehicle" refers to a vehicle capable of implementing at least one navigation change without driver input. A "navigation change" refers to one or more changes in steering, braking, or acceleration of the vehicle. To be autonomous, a vehicle need not be fully automatic (e.g., operating completely without a driver or driver input). Rather, an autonomous vehicle includes a vehicle that can operate under driver control during certain periods of time and without driver control during other periods of time. An autonomous vehicle may also include a vehicle that controls only some aspects of vehicle navigation, such as steering (e.g., to maintain a vehicle course between vehicle lane constraints), but leaves other aspects (e.g., braking) to the driver. In some cases, an autonomous vehicle may handle some or all aspects of braking, speed control, and / or steering of the vehicle.
[0020] Because human drivers typically rely on visual cues and observations to control the vehicle, the transportation infrastructure is built accordingly, with lane markings, traffic signs, and traffic lights all designed to provide visual information to the driver. In light of these design features of the transportation infrastructure, an autonomous vehicle may include a camera and a processing unit that analyzes visual information captured from the vehicle's environment. The visual information may include, for example, components of the transportation infrastructure observable by the driver (e.g., lane markings, traffic signs, traffic lights, etc.) and other obstacles (e.g., other vehicles, pedestrians, debris, etc.). Additionally, an autonomous vehicle may use stored information when navigating, such as information that provides a model of the vehicle's environment. For example, the vehicle may use GPS data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data to provide information related to the vehicle's environment while the vehicle is traveling, and the vehicle (and other vehicles) may use the information to locate itself in the model.
[0021] System Overview FIG. 1 is a block diagram representation of a system 100 according to an exemplary embodiment disclosed. The system 100 may include various components depending on the requirements of a particular implementation. In some embodiments, the system 100 may include a processing unit 110, an image acquisition unit 120, a position sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. The processing unit 110 may include one or more processing devices. In some embodiments, the processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, the image acquisition unit 120 may include any number of image acquisition devices and components depending on the requirements of a particular application. In some embodiments, the image acquisition unit 120 may include one or more image capture devices (e.g., cameras), such as an image capture device 122, an image capture device 124, and an image capture device 126. System 100 may also include a data interface 128 that communicatively connects processing unit 110 to image acquisition unit 120. For example, data interface 128 may include any one or more wired and / or wireless links that transmit image data acquired by image acquisition unit 120 to processing unit 110.
[0022] Wireless transceiver 172 may include one or more devices configured to exchange transmission signals with one or more networks (e.g., cellular, Internet, etc.) over a wireless interface through the use of radio frequencies, infrared frequencies, magnetic fields, or electric fields. Wireless transceiver 172 may transmit and / or receive data using any known standard (e.g., Wi-Fi, Bluetooth, Bluetooth Smart, 802.15.4, ZigBee, etc.).
[0023] Both application processor 180 and image processor 190 may include various types of processing devices. For example, either or both of application processor 180 and image processor 190 may include a microprocessor, a preprocessor (such as an image preprocessor), a graphics processor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, memory, or any other type of device suitable for executing applications and processing and analyzing images. In some embodiments, application processor 180 and / or image processor 190 may include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, etc. A variety of processing devices may be used, including, for example, processors available from manufacturers such as Intel®, AMD®, etc., and may include a variety of architectures (e.g., x86 processor, ARM®, etc.).
[0024] In some embodiments, application processor 180 and / or image processor 190 may include any of the EyeQ series of processors available from Mobileye®. These processor designs include multiple processing units, each with a local memory and instruction set. Such processors may include video inputs that receive image data from multiple image sensors, and may also include video output capabilities. In one example, the EyeQ2® uses 90 nm-micron technology operating at 332 MHz. The EyeQ2® architecture consists of two floating-point hyper-threaded 32-bit RISC CPUs (MIPS32® 34K® cores), five Vision Computation Engines (VCEs), three Vector Microcode Processors (VMPs®), a Denali 64-bit mobile DDR controller, a 128-bit internal acoustic interconnect, dual 16-bit video input and 18-bit video output controllers, a 16-channel DMA, and several peripherals. The MIPS34K CPU manages the five VCEs, three VMPs™ and DMA, a second MIPS34K CPU and multi-channel DMA, and other peripherals. The five VCEs, three VMPs, and the MIPS34K CPU can perform intensive vision calculations required by feature-rich bundle applications. In another example, the disclosed embodiment may use the EyeQ3, a third generation processor that is six times more powerful than the EyeQ2.
[0025] Any processing device disclosed herein may be configured to perform a particular function. Configuring a processing device, such as any of the described EyeQ processors or other controllers or microprocessors, to perform a particular function may include programming computer-executable instructions and providing those instructions to the processing device for execution during operation of the processing device. In some embodiments, configuring a processing device may include programming architectural instructions directly into the processing device. In other embodiments, configuring a processing device may include storing executable instructions in a memory accessible by the processing device during operation. For example, the processing device may access the memory to retrieve and execute the stored instructions during operation.
[0026] 1 shows two separate processing devices included in processing unit 110, more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to accomplish the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by more than two processing devices. Furthermore, in some embodiments, system 100 may include one or more processing units 110 without including other components, such as image acquisition unit 120.
[0027] The processing unit 110 may include various types of devices. For example, the processing unit 110 may include various devices such as a controller, an image preprocessor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, a memory, or any other type of device that processes and analyzes images. The image preprocessor may include a video processor that captures, digitizes, and processes images from an image sensor. The CPU may include any number of microcontrollers or microprocessors. The support circuits may be any number of circuits commonly known in the art, including cache, power, clock, and input / output circuits. The memory may store software that, when executed by the processor, controls the operation of the system. The memory may include databases and image processing software. The memory may include any number of random access memories, read-only memories, flash memories, disk drives, optical storage devices, tape storage devices, removable storage devices, and other types of storage devices. In one example, the memory may be separate from the processing unit 110. In another example, the memory may be integrated into the processing unit 110.
[0028] Each memory 140, 150 may contain software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), may control the operation of various aspects of system 100. These memory units may include various databases and image processing software. The memory units may include random access memory, read-only memory, flash memory, disk drives, optical storage devices, tape storage devices, removable storage devices, and / or any other type of storage device. In some embodiments, memory units 140, 150 may be separate from application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into application processor 180 and / or image processor 190.
[0029] Position sensor 130 may include any type of device suitable for determining a location associated with at least one component of system 100. In some embodiments, position sensor 130 may include a GPS receiver. Such a receiver may determine the location and velocity of a user by processing signals broadcast by Global Positioning System satellites. Position information from position sensor 130 may be provided to application processor 180 and / or image processor 190.
[0030] In some embodiments, the system 100 may include components such as a speed sensor (eg, a tachometer) to measure the speed of the vehicle 200 and / or an accelerometer to measure the acceleration of the vehicle 200.
[0031] User interface 170 may include any device suitable for providing information or receiving input from one or more users of system 100. In some embodiments, user interface 170 may include user input devices including, for example, a touch screen, a microphone, a keyboard, a pointer device, a track wheel, a camera, a knob, buttons, etc. Using such input devices, a user may be able to provide information input or commands to system 100 by typing instructions or information, providing voice commands, selecting on-screen menu options using buttons, pointers, or eye tracking capabilities, or through any other suitable technique for communicating information to system 100.
[0032] User interface 170 may comprise one or more processing devices configured to provide information to or receive information from a user and process that information, e.g., for use by application processor 180. In some embodiments, such processing devices may execute instructions to recognize and track eye movements, receive and interpret voice commands, recognize and interpret touches and / or gestures made on a touchscreen, respond to keyboard entries or menu selections, etc. In some embodiments, user interface 170 may include a display, a speaker, a haptic device, and / or any other device that provides output information to a user.
[0033] Map database 160 may include any type of database that stores map data useful to system 100. In some embodiments, map database 160 may include data related to the locations in a reference coordinate system of various items, including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, etc. Map database 160 may store not only the locations of such items, but also descriptors related to those items, including, for example, names associated with any of the stored features. In some embodiments, map database 160 may be physically located with other components of system 100. Alternatively or additionally, map database 160, or portions thereof, may be located remotely with respect to other components of system 100 (e.g., processing unit 110). In such embodiments, information from map database 160 may be downloaded to a network via a wired or wireless data connection (e.g., via a cellular network and / or the Internet, etc.).
[0034] Image capture devices 122, 124, and 126 may each include any type of device suitable for capturing at least one image from an environment. Furthermore, any number of image capture devices may be used to obtain images for input to the image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, or even four or more image capture devices. Image capture devices 122, 124, and 126 are further described below with reference to Figures 2B-2E.
[0035] System 100 or various components of system 100 may be incorporated into a variety of different platforms. In some embodiments, system 100 may be included in vehicle 200, as shown in FIG. 2A. For example, vehicle 200 may include processing unit 110 and any other components of system 100, as described above with respect to FIG. 1. In some embodiments, vehicle 200 may include only a single image capture device (e.g., a camera), while in other embodiments, such as those discussed in connection with FIGS. 2B-2E, multiple image capture devices may be used. For example, as shown in FIG. 2A, either of image capture devices 122 and 124 of vehicle 200 may be part of an ADAS (advanced driver assistance system) imaging set.
[0036] The image capture device included in vehicle 200 as part of image acquisition unit 120 may be located in any suitable location. In some embodiments, as shown in Figures 2A-2E and 3A-3C, image capture device 122 may be located near the rearview mirror. This location may provide a line of sight similar to that of the driver of vehicle 200 and may assist in determining what the driver can and cannot see. Although image capture device 122 may be located anywhere near the rearview mirror, placing image capture device 122 on the driver's side of the mirror may further assist in capturing an image representative of the driver's field of view and / or line of sight.
[0037] Other locations can also be used for the image capture devices of the image acquisition unit 120. For example, the image capture device 124 can be located on or in the bumper of the vehicle 200. Such a location can be particularly suitable for an image capture device with a wide field of view. The line of sight of an image capture device located in the bumper can be different from the line of sight of the driver, and thus the bumper image capture device and the driver do not always see the same object. The image capture devices (e.g., image capture devices 122, 124, and 126) can also be located in other locations. For example, the image capture devices can be located on one or both side mirrors of the vehicle 200, on the roof of the vehicle 200, on the hood of the vehicle 200, on the trunk of the vehicle 200, on the sides of the vehicle 200, mounted on, positioned behind, or positioned in front of any window of the vehicle 200, mounted on or near the front and / or rear lights of the vehicle 200, etc.
[0038] In addition to the image capture device, vehicle 200 may include various other components of system 100. For example, processing unit 110 may be integrated into the vehicle's engine control unit (ECU) or may be included in vehicle 200 separate from the ECU. Vehicle 200 may also be equipped with a position sensor 130, such as a GPS receiver, and vehicle 200 may also include a map database 160 and memory units 140 and 150.
[0039] As described above, the wireless transceiver 172 may transmit and / or receive data via one or more networks (e.g., a cellular network, the Internet, etc.). For example, the wireless transceiver 172 may upload data collected by the system 100 to one or more servers and download data from one or more servers. Through the wireless transceiver 172, the system 100 may receive updates to the data stored in the map database 160, the memory 140, and / or the memory 150, for example, periodically or on demand. Similarly, the wireless transceiver 172 may upload any data from the system 100 (e.g., images captured by the image acquisition unit 120, data received by the position sensor 130, other sensors, or the vehicle control system, etc.) and / or any data processed by the processing unit 110 to one or more servers.
[0040] System 100 may upload data to a server (e.g., the cloud) based on a privacy level setting. For example, system 100 may implement a privacy level setting that regulates or limits the types of data (including metadata) that may uniquely identify the vehicle and / or the driver / owner of the vehicle that are sent to the server. Such settings may be set by a user via wireless transceiver 172, initialized by factory default settings, or set by data received by wireless transceiver 172, for example.
[0041] In some embodiments, system 100 may upload data according to a "high" privacy level, and under certain settings, system 100 may transmit data (e.g., location information associated with a route, captured images, etc.) without any details about a particular vehicle and / or driver / owner. For example, when uploading data according to a "high" privacy level, system 100 may transmit data such as captured images and / or limited location information associated with a route that does not include the vehicle identification number (VIN) or the name of the driver or owner of the vehicle, but instead.
[0042] Other privacy levels are contemplated. For example, system 100 may transmit data to the server according to a "medium" privacy level, which may include additional information not included under a "high" privacy level, such as the make and / or model of the vehicle and / or vehicle type (e.g., passenger car, sport utility vehicle, truck, etc.). In some embodiments, system 100 may upload data according to a "low" privacy level. Under a "low" privacy level setting, system 100 may upload and include sufficient data to uniquely identify a particular vehicle, owner / driver, and / or some or all of the route traveled by the vehicle. Such "low" privacy level data may include, for example, one or more of the VIN, driver / owner name, starting point of the vehicle prior to departure, intended destination of the vehicle, make and / or model of the vehicle, type of vehicle, etc.
[0043] FIG 2A is a side view representation of an exemplary vehicle imaging system according to a disclosed embodiment. FIG 2B is a top view representation of the embodiment shown in FIG 2A. As shown in FIG 2B, the disclosed embodiment may show a vehicle 200 including a system 100 within the body having a first image capture device 122 positioned near a rearview mirror and / or near a driver of the vehicle 200, a second image capture device 124 positioned on or within a bumper area (e.g., one of the bumper areas 210) of the vehicle 200, and a processing unit 110.
[0044] As shown in Figure 2C, image capture devices 122 and 124 may both be positioned near the rearview mirror and / or near the driver of vehicle 200. Additionally, while two image capture devices 122 and 124 are shown in Figures 2B and 2C, it should be understood that other embodiments may include three or more image capture devices. For example, in the embodiment shown in Figures 2D and 2E, a first image capture device 122, a second image capture device 124, and a third image capture device 126 are included in system 100 of vehicle 200.
[0045] 2D, image capture device 122 may be positioned near a rearview mirror and / or near the driver of vehicle 200, and image capture devices 124 and 126 may be positioned on a bumper area (e.g., one of bumper areas 210) of vehicle 200. Also, as shown in FIG. 2E, image capture devices 122, 124, and 126 may be positioned near a rearview mirror and / or near the driver's seat of vehicle 200. The disclosed embodiments are not limited to any particular number and configuration of image capture devices, and image capture devices may be positioned in any suitable location within and / or on vehicle 200.
[0046] It should be understood that the disclosed embodiments are not limited to vehicles and may be applicable in other contexts, and it should also be understood that the disclosed embodiments are not limited to a particular type of vehicle 200 and may be applicable to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.
[0047] The first image capture device 122 may include any suitable type of image capture device. The image capture device 122 may include an optical axis. In one example, the image capture device 122 may include an Aptina M9V024 WVGA sensor with a global shutter. In other embodiments, the image capture device 122 may provide a resolution of 1280×960 pixels and may include a rolling shutter. The image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included to provide, for example, a desired focal length and field of view for the image capture device. In some embodiments, a 6 mm lens or a 12 mm lens may be associated with the image capture device 122. In some embodiments, the image capture device 122 may be configured to capture an image having a desired field of view (FOV) 202, as shown in FIG. 2D. For example, image capture device 122 may be configured to have a regular FOV, such as in the range of 40 degrees to 56 degrees, including a 46 degree FOV, a 50 degree FOV, a 52 degree FOV, or degrees greater than 52 degrees. Alternatively, image capture device 122 may be configured to have a narrow FOV, such as in the range of 23 degrees to 40 degrees, such as a 28 degree FOV or a 36 degree FOV. In addition, image capture device 122 may be configured to have a wide FOV, such as in the range of 100 degrees to 180 degrees. In some embodiments, image capture device 122 may include a wide angle bumper camera or a bumper camera with up to 180 degrees FOV. In some embodiments, image capture device 122 may be a 7.2 Mpixel image capture device with an aspect ratio of about 2:1 (e.g., H×V=3800×1900 pixels) with a horizontal FOV of about 100 degrees. Such an image capture device may be used in place of a three-dimensional image capture device configuration. Due to large lens distortion, the vertical FOV of such an image capture device can be much less than 50 degrees in implementations in which the image capture device uses a radially symmetric lens, for example, such lenses are not radially symmetric, thereby allowing a vertical FOV greater than 50 degrees with a horizontal FOV of 100 degrees.
[0048] The first image capture device 122 may acquire a plurality of first images of a scene associated with the vehicle 200. The plurality of first images may each be acquired as a series of image scan lines, which may be captured using a rolling shutter. Each scan line may include a plurality of pixels.
[0049] The first image capture device 122 may have a scan rate associated with the acquisition of each of the first series of image scan lines. The scan rate may refer to the rate at which the image sensor can acquire image data associated with each pixel included in a particular scan line.
[0050] Image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, including, for example, CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be utilized with a rolling shutter whereby each pixel in a row is read one at a time, and the scanning of the rows proceeds row by row until the entire image frame is captured. In some embodiments, the rows may be captured sequentially from top to bottom for the frame.
[0051] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124, and 126) may constitute a high-resolution imager and may have a resolution of greater than 5M pixels, greater than 7M pixels, greater than 10M pixels, or more.
[0052] The use of a rolling shutter can result in pixels in different rows being exposed and captured at different times, which can cause skew and other image artifacts in the captured image frame. On the other hand, if the image capture device 122 is configured to operate with a global or synchronous shutter, all pixels can be exposed for the same amount of time during a common exposure period. As a result, image data in a frame collected from a system utilizing a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a particular time. Conversely, when applying a rolling shutter, each row in a frame is exposed and data is captured at a different time. Thus, moving objects can appear distorted with an image capture device having a rolling shutter. This phenomenon is described in more detail below.
[0053] The second image capture device 124 and the third image capture device 126 may be any type of image capture device. Like the first image capture device 122, each of the image capture devices 124 and 126 may include an optical axis. In one embodiment, each of the image capture devices 124 and 126 may include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of the image capture devices 124 and 126 may include a rolling shutter. Like the image capture device 122, the image capture devices 124 and 126 may be configured to include various lenses and optical elements. In some embodiments, the lenses associated with the image capture devices 124 and 126 may provide the same FOV (such as FOV 202) as the FOV associated with the image capture device 122, or a narrower FOV (such as FOVs 204 and 206). For example, image capture devices 124 and 126 may have an FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less than 20 degrees.
[0054] Image capture devices 124 and 126 may acquire multiple second and third images of a scene associated with vehicle 200. Each of the multiple second and third images may be acquired as second and third series of image scan lines, which may be captured using a rolling shutter. Each scan line or row may have a number of pixels. Image capture devices 124 and 126 may have second and third scan rates associated with acquiring each of the second and third series of image scan lines.
[0055] Each image capture device 122, 124, and 126 may be positioned in any suitable location and in any suitable orientation relative to vehicle 200. The relative positions of image capture devices 122, 124, and 126 may be selected to assist in fusing together information obtained from the image capture devices. For example, in some embodiments, the FOV associated with image capture device 124 (FOV 204) may partially or completely overlap with the FOV associated with image capture device 122 (e.g., FOV 202) and the FOV associated with image capture device 126 (e.g., FOV 206).
[0056] Image capture devices 122, 124, and 126 may be positioned on vehicle 200 at any suitable relative height. In one example, there may be a height difference between image capture devices 122, 124, and 126, which may provide sufficient parallax information to enable stereo analysis. For example, as shown in FIG. 2A, the two image capture devices 122 and 124 are at different heights. There may also be a lateral displacement difference between image capture devices 122, 124, and 126, which may provide additional parallax information for stereo analysis by processing unit 110, for example. The lateral displacement difference may be represented as d xIn some embodiments, a forward or rearward displacement (e.g., range displacement) may exist between image capture devices 122, 124, 126. For example, image capture device 122 may be positioned 0.5 to 2 meters or more behind image capture device 124 and / or image capture device 126. With this type of displacement, one of the image capture devices may be able to cover a potential blind spot of the other image capture device.
[0057] Image capture device 122 may have any suitable resolution capability (e.g., the number of pixels associated with the image sensor), and the resolution of the image sensor associated with image capture device 122 may be higher, lower, or the same as the resolution of the image sensors associated with image capture devices 124 and 126. In some embodiments, the image sensors associated with image capture device 122 and / or image capture devices 124 and 126 may have a resolution of 640×480, 1024×768, 1280×960, or any other suitable resolution.
[0058] The frame rate (e.g., the rate at which the image capture device acquires a set of pixel data for one image frame before moving on to capture pixel data associated with the next image frame) may be controllable. The frame rate associated with image capture device 122 may be higher, lower, or the same as the frame rate associated with image capture devices 124 and 126. The frame rates associated with image capture devices 122, 124, and 126 may depend on a variety of factors that may affect the timing of the frame rate. For example, one or more of image capture devices 122, 124, and 126 may include a selectable pixel delay period imposed before or after acquisition of image data associated with one or more pixels of an image sensor in image capture devices 122, 124, and / or 126. In general, image data corresponding to each pixel may be acquired according to the device's clock rate (e.g., one pixel per clock cycle). Additionally, in embodiments including a rolling shutter, one or more of image capture devices 122, 124, and 126 may include a selectable horizontal blanking period that is imposed before or after acquisition of image data associated with a row of pixels of an image sensor in image capture devices 122, 124, and / or 126. Additionally, one or more of image capture devices 122, 124, and / or 126 may include a selectable vertical blanking period that is imposed before or after acquisition of image data associated with an image frame of image capture devices 122, 124, and 126.
[0059] These timing controls may enable the frame rates associated with image capture devices 122, 124, and 126 to be synchronized even if the line scan rate of each image capture device is different. Additionally, as discussed in more detail below, among other factors (e.g., image sensor resolution, maximum line scan rate, etc.), these selectable timing controls may enable the synchronization of image capture from areas where the FOV of image capture device 122 overlaps with one or more of the FOVs of image capture devices 124 and 126, even if the field of view of image capture device 122 differs from the FOV of image capture devices 124 and 126.
[0060] Frame rate timing at image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensor. For example, assuming both devices have similar line scan rates, if one device includes an image sensor with a resolution of 640×480 and the other device includes an image sensor with a resolution of 1280×960, capturing a frame of image data from the sensor with the higher resolution will require a longer period of time.
[0061] Another factor that may affect the timing of image data acquisition at image capture devices 122, 124, and 126 is the maximum line scan rate. For example, acquisition of a row of image data from an image sensor included in image capture devices 122, 124, and 126 requires some minimum amount of time. Assuming no pixel delay period is added, this minimum amount of time to acquire a row of image data will be related to the maximum line scan rate of a particular device. Devices that offer higher maximum line scan rates have the potential to provide higher frame rates than devices that have lower maximum line scan rates. In some embodiments, one or both of image capture devices 124 and 126 may have a maximum line scan rate that is higher than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture devices 124 and / or 126 may be 1.25 times, 1.5 times, 1.75 times, or 2 times or more than the maximum line scan rate of image capture device 122.
[0062] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may operate at a scan rate that is equal to or less than its maximum scan rate. The system may be configured such that one or both of image capture devices 124 and 126 operate at a line scan rate that is equal to the line scan rate of image capture device 122. In other examples, the system may be configured such that the line scan rate of image capture devices 124 and / or 126 may be 1.25 times, 1.5 times, 1.75 times, or 2 times or more the line scan rate of image capture device 122.
[0063] In some embodiments, image capture devices 122, 124, and 126 may be asymmetric. That is, the image capture devices may include cameras with different fields of view (FOV) and focal lengths. The fields of view of image capture devices 122, 124, and 126 may include any desired area of the environment of vehicle 200, for example. In some embodiments, one or more of image capture devices 122, 124, and 126 may be configured to acquire image data from the environment in front of vehicle 200, the environment behind vehicle 200, the environment on either side of vehicle 200, or a combination thereof.
[0064] Further, the focal length associated with each image capture device 122, 124, and / or 126 may be selectable (e.g., by inclusion of an appropriate lens, etc.) such that each device captures images of objects at a desired distance range from vehicle 200. For example, in some embodiments, image capture devices 122, 124, and 126 may capture images of close-by objects within a few meters of the vehicle. Image capture devices 122, 124, 126 may also be configured to capture images of objects at greater distances from the vehicle (e.g., 25 m, 50 m, 100 m, 150 m, or more). Further, the focal lengths of image capture devices 122, 124, and 126 may be selected such that one image capture device (e.g., image capture device 122) can capture images of objects relatively close to the vehicle (e.g., within 10 m or within 20 m), while the other image capture devices (e.g., image capture devices 124 and 126) can capture images of objects farther away from vehicle 200 (e.g., greater than 20 m, greater than 50 m, greater than 100 m, greater than 150 m, etc.).
[0065] According to some embodiments, the FOV of one or more of image capture devices 122, 124, and 126 may have a wide angle. For example, it may be advantageous to have an FOV of 140 degrees, particularly for image capture devices 122, 124, and 126 that may be used to capture images of areas near vehicle 200. For example, image capture device 122 may be used to capture images of areas to the right or left of vehicle 200, and in such embodiments, it may be desirable for image capture device 122 to have a wide FOV (e.g., at least 140 degrees).
[0066] The field of view associated with each of the image capture devices 122, 124, and 126 may depend on the respective focal lengths. For example, as the focal length increases, the corresponding field of view decreases.
[0067] Image capture devices 122, 124, and 126 may be configured to have any suitable field of view. In one particular example, image capture device 122 may have a horizontal FOV of 46 degrees, image capture device 124 may have a horizontal FOV of 23 degrees, and image capture device 126 may have a horizontal FOV between 23 and 46 degrees. In another example, image capture device 122 may have a horizontal FOV of 52 degrees, image capture device 124 may have a horizontal FOV of 26 degrees, and image capture device 126 may have a horizontal FOV between 26 and 52 degrees. In some embodiments, the ratio of the FOV of image capture device 122 to the FOV of image capture device 124 and / or image capture device 126 may vary between 1.5 and 2.0. In other embodiments, the ratio may vary between 1.25 and 2.25.
[0068] System 100 may be configured such that the field of view of image capture device 122 at least partially or completely overlaps the field of view of image capture device 124 and / or image capture device 126. In some embodiments, system 100 may be configured such that the fields of view of image capture devices 124 and 126, for example, fall within the field of view of image capture device 122 (e.g., are smaller than the field of view of image capture device 122) and share a common center with the field of view of image capture device 122. In other embodiments, image capture devices 122, 124, and 126 may capture adjacent FOVs or may have partially overlapping FOVs. In some embodiments, the fields of view of image capture devices 122, 124, and 126 may be aligned such that the center of image capture device 124 and / or 126, which has a narrower FOV, may be located in the bottom half of the field of view of device 122, which has a wider FOV.
[0069] 2F is a diagrammatic representation of an exemplary vehicle control system according to disclosed embodiments. As shown in FIG. 2F, vehicle 200 may include a throttle system 220, a brake system 230, and a steering system 240. System 100 may provide inputs (e.g., control signals) to one or more of throttle system 220, brake system 230, and steering system 240 via one or more data links (e.g., any wired and / or wireless link or link that transmits data). For example, based on analysis of images acquired by image capture devices 122, 124, and / or 126, system 100 may provide control signals to one or more of throttle system 220, brake system 230, and steering system 240 to navigate vehicle 200 (e.g., by causing it to accelerate, turn, lane shift, etc.). Additionally, system 100 may receive inputs indicative of operating conditions of vehicle 200 (e.g., speed, whether vehicle 200 is braking and / or turning, etc.) from one or more of throttle system 220, braking system 230, and steering system 24. Further details are provided below in conjunction with Figures 4-7.
[0070] As shown in FIG. 3A , the vehicle 200 may also include a user interface 170 for interacting with a driver or passenger of the vehicle 200. For example, the user interface 170 in a vehicle application may include a touch screen 320, knobs 330, buttons 340, and a microphone 350. The driver or passenger of the vehicle 200 may also interact with the system 100 using a steering wheel (e.g., located on or near the steering column of the vehicle 200, including, for example, a turn signal handle), buttons (e.g., located on the steering wheel of the vehicle 200), and the like. In some embodiments, the microphone 350 may be positioned adjacent to the rearview mirror 310. Similarly, in some embodiments, the image capture device 122 may be located near the rearview mirror 310. In some embodiments, the user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, the system 100 may provide various notifications (e.g., alerts) via the speaker 360.
[0071] 3B-3D are diagrams of an exemplary camera mount 370 configured to be positioned behind a rearview mirror (e.g., rearview mirror 310) and opposite a vehicle windshield, according to disclosed embodiments. As shown in FIG. 3B, camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be positioned behind glare shield 380, which may directly contact the windshield and may include a film and / or composition of anti-reflective materials. For example, glare shield 380 may be positioned to be aligned opposite a windshield having a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 may be positioned behind glare shield 380, for example, as shown in FIG. 3D. The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124, and 126, camera mount 370, and glare shield 380. FIG. 3C is a view of the camera mount 370 shown in FIG. 3B from the front.
[0072] As will be appreciated by one of ordinary skill in the art having the benefit of this disclosure, many variations and / or modifications may be made to the above disclosed embodiments. For example, not all components are essential to the operation of system 100. Furthermore, any component may be located in any suitable portion of system 100, and the components may be rearranged in various configurations while still providing the functionality of the disclosed embodiments. Thus, the above-described configurations are examples, and regardless of the above-described configuration, system 100 may provide a wide range of functionality for analyzing the surroundings of vehicle 200 and navigating vehicle 200 in response to the analysis.
[0073] As discussed in more detail below, in accordance with various disclosed embodiments, system 100 may provide various features related to autonomous driving and / or driver assistance technologies. For example, system 100 may analyze image data, location data (e.g., GPS location information), map data, speed data, and / or data from sensors included in vehicle 200. System 100 may collect data for analysis, for example, from image capture unit 120, location sensor 130, and other sensors. Additionally, system 100 may analyze the collected data to identify whether vehicle 200 should take a particular action, and then automatically take the determined action without human intervention. For example, when vehicle 200 navigates without human participation, system 100 may automatically control braking, acceleration, and / or steering of vehicle 200 (e.g., by sending control signals to one or more of throttle system 220, braking system 230, and steering system 240). Additionally, system 100 may analyze the collected data and issue warnings and / or alerts to vehicle occupants based on the analysis of the collected data. Further details regarding various embodiments provided by system 100 are provided below.
[0074] Forward-looking multi-imaging system As described above, the system 100 may provide a driver assistance function using a multi-camera system. The multi-camera system may use one or more cameras facing in a forward direction of the vehicle. In other embodiments, the multi-camera system may include one or more cameras facing the side of the vehicle or the rear of the vehicle. In one embodiment, for example, the system 100 may use a two-camera imaging system, in which a first camera and a second camera (e.g., image capture devices 122 and 124) may be positioned at the front and / or side of the vehicle (e.g., vehicle 200). The first camera may have a field of view that is larger, smaller, or partially overlaps the field of view of the second camera. Furthermore, the first camera may be connected to a first image processor to perform monocular image analysis of the image provided by the first camera, and the second camera may be connected to a second image processor to perform monocular image analysis of the image provided by the second camera. The outputs (e.g., processed information) of the first and second image processors may be combined. In some embodiments, the second image processor may receive images from both the first and second cameras to perform stereo analysis. In another embodiment, the system 100 may use a three-camera imaging system, where each camera has a different field of view. Thus, such a system may make decisions based on information derived from objects at various distances both in front of and to the sides of the vehicle. References to monocular image analysis may refer to when image analysis is performed based on images captured from a single viewpoint (e.g., a single camera). Stereo image analysis may refer to when image analysis is performed based on two or more images captured with one or more of the image capture parameters changed. For example, captured images suitable for performing stereo image analysis may include images captured from two or more different positions, images captured from different fields of view, images captured using different focal lengths, images captured with parallax information, etc.
[0075] For example, in one embodiment, system 100 may implement a three camera configuration using image capture devices 122-126. In such a configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees or other value selected from the range of about 20-45 degrees, etc.), image capture device 124 may provide a wide field of view (e.g., 150 degrees or other value selected from the range of about 100-180 degrees), and image capture device 126 may provide a medium field of view (e.g., 46 degrees or other value selected from the range of about 35-60 degrees). In some embodiments, image capture device 126 may operate as a main or primary camera. Image capture devices 122-126 may be positioned substantially side-by-side (e.g., 6 cm apart) behind rearview mirror 310. Additionally, in some embodiments, as described above, one or more of image capture devices 122-126 may be mounted behind glare shield 380 that is flush with the windshield of vehicle 200. Such a shield may operate to minimize the effect of any reflections from the interior of the vehicle on the image capture devices 122-126.
[0076] 3B and 3C, the wide field of view camera (e.g., image capture device 124 in the example above) may be mounted lower than the narrow main field of view camera (e.g., image capture devices 122 and 126 in the example above). This configuration may provide a free line of sight from the wide field of view camera. To reduce reflections, the camera may be mounted near the windshield of vehicle 200 and may include a polarizer to attenuate reflected light.
[0077] A three camera system may provide certain performance characteristics. For example, some embodiments may include the ability to verify the detection of an object by one camera based on the detection results from another camera. In the three camera configuration described above, processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips as described above), each processing device dedicated to processing images captured by one or more of image capture devices 122-126.
[0078] In a three-camera system, the first processing device may receive images from both the main camera and the narrow FOV camera and may perform vision processing of the narrow FOV camera to detect, for example, other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the first processing device may calculate pixel discrepancies between the images from the main camera and the narrow camera and create a 3D reconstruction of the vehicle 200's environment. The first processing device may then combine the 3D reconstruction with 3D map data or 3D information calculated based on information from another camera.
[0079] The second processing device may receive images from the main camera and perform vision processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the second processing device may calculate camera displacement, calculate pixel discrepancies between successive images based on the displacement, and create a 3D reconstruction (e.g., structure-from-motion) of the scene. The second processing device may send the structure-from-motion based on the 3D reconstruction to the first processing device and combine the structure-from-motion with the stereoscopic 3D image.
[0080] The third processing device may receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device may further execute additional processing instructions to analyze the images and identify moving objects in the images, such as vehicles changing lanes, pedestrians, etc.
[0081] In some embodiments, having image-based information streams captured and processed independently may provide an opportunity to provide redundancy in the system, such as using a first image capture device and images processed from that device to verify and / or capture information obtained by capturing and processing image information from at least a second image capture device.
[0082] In some embodiments, system 100 may use two image capture devices (e.g., image capture devices 122 and 124) in providing navigation assistance to vehicle 200, and a third image capture device (e.g., image capture device 126) may be used to provide redundancy and verify analysis of data received from the other two image capture devices. For example, in such a configuration, image capture devices 122 and 124 may provide images for stereo analysis by system 100 for navigating vehicle 200, while image capture device 126 may provide images for monocular analysis by system 100 to provide redundancy and verification of information derived based on images captured from image capture device 123 and / or image capture device 124. That is, image capture device 126 (and corresponding processing device) may be viewed as providing a redundant subsystem that provides a check on the analysis derived from image capture devices 122 and 124 (e.g., providing an automatic emergency braking (AEB) system).
[0083] Those skilled in the art will recognize that the above camera configurations, camera placements, camera numbers, camera locations, etc. are merely exemplary. These components, etc. described for an overall system, may be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of multi-camera systems to provide driver assistance and / or autonomous vehicle functions follow below.
[0084] 4 is an example functional block diagram of memory 140 and / or 150 that may be stored / programmed with instructions to perform one or more operations in accordance with the disclosed embodiments. Although reference is made below to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.
[0085] As shown in FIG. 4, memory 140 may store monocular image analysis module 402, stereo image analysis module 404, velocity and acceleration module 406, and navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 402-408 included in memory 140. Those skilled in the art will appreciate that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Thus, any steps of the following processes may be performed by one or more processing devices.
[0086] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform monocular image analysis of a set of images acquired by one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensor information (e.g., information from radar) to perform the monocular image analysis. As described in connection with FIGS. 5A-5D below, the monocular image analysis module 402 may include instructions to detect a set of features in the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazards, and any other features associated with the vehicle's environment. Based on the analysis, the system 100 may cause one or more navigational responses in the vehicle 200 (e.g., via the processing unit 110), such as turns, lane shifts, and acceleration changes, as discussed below in connection with the navigation response module 408.
[0087] In one embodiment, stereo image analysis module 404 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, perform stereo image analysis of a first and second set of images acquired by a combination of image capture devices selected from image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from the first and second sets of images with additional sensor information (e.g., information from radar) to perform stereo image analysis. For example, stereo image analysis module 404 may include instructions to perform stereo image analysis based on a first set of images acquired by image capture device 124 and a second set of images acquired by image capture device 126. As described below in connection with FIG. 6, stereo image analysis module 404 may include instructions to detect a set of features in the first and second sets of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and hazards. Based on the analysis, the processing unit 110 may cause one or more navigational responses in the vehicle 200, such as turns, lane shifts, and acceleration changes, as described below in connection with the navigational response module 408.
[0088] In one embodiment, the speed and acceleration module 406 may store software configured to analyze data received from one or more computational and electromechanical devices in the vehicle 200 configured to alter the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 may execute instructions associated with the speed and acceleration module 406 to calculate a target speed of the vehicle 200 based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include, for example, target position, speed, and / or acceleration, the position and / or speed of the vehicle 200 relative to nearby vehicles, pedestrians, or road objects, and position information of the vehicle 200 relative to lane markings on the road. Additionally, the processing unit 110 may calculate a target speed of the vehicle 200 based on sensor inputs (e.g., information from a radar) and inputs from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and / or the steering system 240 of the vehicle 200. Based on the calculated target speed, the processing unit 110 may send electronic signals to the throttle system 220, the braking system 230, and / or the steering system 240 of the vehicle 200 to trigger a change in speed and / or acceleration, for example, by physically reducing the brakes or reducing the accelerator of the vehicle 200.
[0089] In one embodiment, the navigation response module 408 may store software executable by the processing unit 110 to determine a desired navigation response based on data derived from execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include position and speed information associated with nearby vehicles, pedestrians, and road objects, as well as target position information for the vehicle 200, and the like. Additionally, in some embodiments, the navigation response may be based (partially or fully) on map data, a predetermined position of the vehicle 200, and / or a relative speed or relative acceleration between the vehicle 200 and one or more objects detected from execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 may also determine a desired navigation response based on sensor inputs (e.g., information from a radar) and inputs from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and the steering system 240 of the vehicle 200. Based on the desired navigational response, processing unit 110 may trigger the desired navigational response by sending electronic signals to throttle system 220, braking system 230, and steering system 240 of vehicle 200 to, for example, turn the steering wheel of vehicle 200 to achieve a predetermined angle of rotation. In some embodiments, processing unit 110 may use the output of navigational response module 408 (e.g., the desired navigational response) as an input to the execution of velocity and acceleration module 406 to calculate a speed change for vehicle 200.
[0090] FIG. 5A is a flow chart illustrating an example process 500A for generating one or more navigational responses based on monocular image analysis, according to disclosed embodiments. In step 510, processing unit 110 may receive a plurality of images via data interface 128 between processing unit 110 and image acquisition unit 120. For example, a camera (such as image capture device 122 having field of view 202) included in image acquisition unit 120 may capture a plurality of images of an area in front of vehicle 200 (or, for example, to the side or rear of the vehicle) and transmit them to processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). Processing unit 110 may execute monocular image analysis module 402 to analyze the plurality of images in step 520, as described in further detail below in connection with FIGS. 5B-5D. By performing the analysis, processing unit 110 may detect a set of features in the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, and traffic lights.
[0091] Processing unit 110 may also execute monocular image analysis module 402 in step 520 to detect various road hazards, such as, for example, truck tire parts, fallen road signs, loose cargo, and small animals. Road hazards may vary in structure, shape, size, and color, making such hazards more difficult to detect. In some embodiments, processing unit 110 may execute monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, processing unit 110 may estimate camera movement between consecutive image frames and calculate pixel discrepancies between frames to build a 3D map of the road. Processing unit 110 may then use the 3D map to detect the road surface and hazards present on the road surface.
[0092] In step 530, processing unit 110 may execute navigation response module 408 to cause vehicle 200 to generate one or more navigation responses based on the analysis performed in step 520 and the techniques described above in connection with FIG. 4. The navigation responses may include, for example, a turn, a lane shift, an acceleration change, and the like. In some embodiments, processing unit 110 may use data derived from execution of speed and acceleration module 406 to generate one or more navigation responses. Furthermore, multiple navigation responses may occur simultaneously, sequentially, or any combination thereof. For example, processing unit 110 may cause vehicle 200 to cross a lane and then, for example, accelerate, for example, by sequentially sending control signals to steering system 240 and throttle system 220 of vehicle 200. Alternatively, processing unit 110 may cause vehicle 200 to brake and simultaneously shift lanes, for example, by simultaneously sending control signals to braking system 230 and steering system 240 of vehicle 200.
[0093] FIG. 5B is a flow chart illustrating an example process 500B for detecting one or more vehicles and / or pedestrians in a set of images according to the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500B. In step 540, processing unit 110 may identify a set of candidate objects representing vehicles and / or pedestrians that may be present. For example, processing unit 110 may scan one or more images, compare the images to one or more predefined patterns, and identify locations within each image that may contain a target object (e.g., a vehicle, a pedestrian, or a portion thereof). The predefined patterns may be specified to achieve a high rate of "false hits" and a low rate of "misses." For example, processing unit 110 may use a low similarity threshold to the predefined patterns to identify candidate objects as possible vehicles or pedestrians. By doing so, processing unit 110 may be able to reduce the probability of missing (e.g., not identifying) a candidate object representing a vehicle or a pedestrian.
[0094] At step 542, processing unit 110 may filter the set of candidate objects to exclude certain candidates (e.g., irrelevant or less relevant objects) based on classification criteria. Such criteria may be derived from various characteristics associated with object types stored in a database (e.g., a database stored in memory 140). The characteristics may include the object's shape, dimensions, texture, and position (e.g., relative to vehicle 200), etc. Thus, processing unit 110 may use one or more sets of criteria to reject false candidates from the set of candidate objects.
[0095] At step 544, processing unit 110 may analyze multiple image frames to identify whether objects in the set of candidate images represent vehicles and / or pedestrians. For example, processing unit 110 may track detected candidate objects across successive frames and accumulate per-frame data associated with the detected objects (e.g., size, position relative to vehicle 200, etc.). Additionally, processing unit 110 may estimate parameters of the detected objects and compare the per-frame position data of the objects to predicted positions.
[0096] In step 546, processing unit 110 may construct a set of measurements of the detected objects. Such measurements may include, for example, position, velocity, and acceleration values (relative to vehicle 200) associated with the detected objects. In some embodiments, processing unit 110 may construct the measurements based on estimation techniques using a series of time-based observations, such as a Kalman filter or linear quadratic estimation (LQE), and / or modeling data available for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). A Kalman filter may be based on a measurement of the scale of the object, where the scale measurement is proportional to the time to impact (e.g., the amount of time it takes vehicle 200 to reach the object). Thus, by performing steps 540-546, processing unit 110 may identify vehicles and pedestrians appearing in the set of captured images and derive information (e.g., position, velocity, size) associated with the vehicles and pedestrians. Based on the identification and derived information, processing unit 110 may cause one or more navigation responses in vehicle 200, as described above in connection with FIG. 5A.
[0097] In step 548, processing unit 110 may perform optical flow analysis of one or more images to reduce the probability of detecting "false hits" and the probability of missing candidate objects representing vehicles or pedestrians. Optical flow analysis may refer to, for example, analyzing movement patterns, separate from road surface movement, for vehicle 200 in one or more images associated with other vehicles and pedestrians. Processing unit 110 may calculate the movement of candidate objects by observing different positions of the object across multiple image frames captured at different times. Processing unit 110 may use the position and time values as inputs to a mathematical model to calculate the movement of candidate objects. Thus, optical flow analysis may provide another way to detect vehicles and pedestrians in the vicinity of vehicle 200. Processing unit 110 may perform optical flow analysis in combination with steps 540-546 to provide redundancy for detecting vehicles and pedestrians and to increase the reliability of system 100.
[0098] FIG. 5C is a flow chart illustrating an example process 500C for detecting road marks and / or lane geometry information in a set of images according to the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500C. In step 550, processing unit 110 may detect a set of objects by scanning one or more images. To detect lane mark segments, lane geometry information, and other relevant road marks, processing unit 110 may filter the set of objects to exclude those determined to be irrelevant (e.g., small holes, small rocks, etc.). In step 552, processing unit 110 may group together the segments detected in step 550 that belong to the same road mark or lane mark. Based on the grouping, processing unit 110 may develop a model, such as a mathematical model, to represent the detected segments.
[0099] In step 554, processing unit 110 may construct a set of measurements associated with the detected segment. In some embodiments, processing unit 110 may create a projection of the detected segment from the image plane to the real-world plane. The projection may be characterized using a third-order polynomial with coefficients corresponding to physical properties such as position, slope, curvature, and curvature derivatives of the detected road. In generating the projection, processing unit 110 may take into account road surface changes and pitch and roll rates associated with vehicle 200. In addition, processing unit 110 may model the road height by analyzing the position and motion cues present on the road surface. Furthermore, processing unit 110 may estimate pitch and roll rates associated with vehicle 200 by tracking a set of feature points in one or more images.
[0100] In step 556, processing unit 110 may perform a multi-frame analysis, for example, by tracking the detected segment across successive image frames and accumulating frame-by-frame data associated with the detected segment. When processing unit 110 performs a multi-frame analysis, the set of measurements constructed in step 554 may become more reliable and may be associated with an increasingly higher degree of confidence. Thus, by performing steps 550-556, processing unit 110 may identify road marks appearing in the set of captured images and derive lane geometry information. Based on the identification and derived information, processing unit 110 may cause one or more navigational responses in vehicle 200, as described above in connection with FIG. 5A.
[0101] In step 558, processing unit 110 may further develop a safety model of vehicle 200 in the vehicle's surroundings, taking into account additional information sources. Processing unit 110 may use the safety model to define situations in which system 100 may safely execute autonomous control of vehicle 200. To develop the safety model, in some embodiments, processing unit 110 may consider the positions and movements of other vehicles, detected road edges and barriers, and / or general road shape descriptions extracted from map data (such as data from map database 160). By taking into account additional information sources, processing unit 110 may provide redundancy for detecting road marks and lane geometry and increase the reliability of system 100.
[0102] FIG. 5D is a flow chart illustrating an example process 500D for detecting traffic lights in a set of images according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500D. In step 560, processing unit 110 may scan the set of images and identify objects that appear at locations in the images that are likely to contain traffic lights. For example, processing unit 110 may filter the identified objects to build a set of candidate objects that excludes objects that are unlikely to correspond to traffic lights. The filtering may be based on various characteristics associated with traffic lights, such as shape, size, texture, and location (e.g., relative to vehicle 200). Such characteristics may be based on many examples of traffic lights and traffic control signals and may be stored in a database. In some embodiments, processing unit 110 may perform a multi-frame analysis on the set of candidate objects that may be traffic lights. For example, processing unit 110 may track the candidate objects across consecutive image frames, estimate the real-world positions of the candidate objects, and filter out objects that are moving (and therefore unlikely to be traffic lights). In some embodiments, processing unit 110 may perform color analysis on the candidate objects to identify the relative location of the detected color represented within the candidate object, which may be a traffic light.
[0103] In step 562, processing unit 110 may analyze the geometry of the intersection. The analysis may be based on any combination of (i) the number of lanes detected on either side of vehicle 200, (ii) marks (such as arrow marks) detected on the road, and (iii) a description of the intersection extracted from map data (such as data from map database 160). Processing unit 110 may perform the analysis using information derived from execution of monocular analysis module 402. In addition, processing unit 110 may identify a correspondence between traffic lights detected in step 560 and lanes appearing in the vicinity of vehicle 200.
[0104] As vehicle 200 approaches the intersection, in step 564, processing unit 110 may update a confidence associated with the analyzed intersection geometry and the detected traffic lights. For example, the number of traffic lights estimated to appear at the intersection compared to the number that actually appear at the intersection may affect the confidence. Thus, based on the confidence, processing unit 110 may delegate control to the driver of vehicle 200 to improve safety conditions. By performing steps 560-564, processing unit 110 may identify traffic lights appearing in the set of captured images and analyze the intersection geometry information. Based on the identification and analysis, processing unit 110 may cause one or more navigation responses in vehicle 200, as described above in connection with FIG. 5A.
[0105] 5E is a flowchart of an example process 500E for generating one or more navigation responses in vehicle 200 based on a vehicle path, according to disclosed embodiments. In step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. The vehicle path may be represented using a set of points represented by coordinates (x,y), with distances d between two points in the set of points being expressed as i may be in the range of 1 to 5 meters. In one embodiment, processing unit 110 may construct an initial vehicle path using two polynomials, such as left and right road polynomials. Processing unit 110 may calculate the geometric midpoint between the two polynomials and, if there is a predefined offset (offset 0 may correspond to driving in the center of the lane), offset each point in the resulting vehicle path by the predefined offset (e.g., smart lane offset). The offset may be in a direction perpendicular to the segment between any two points in the vehicle path. In another embodiment, processing unit 110 may use one polynomial and an estimated lane width to offset each point in the vehicle path by half the estimated lane width plus a predefined offset (e.g., smart lane offset).
[0106] In step 572, processing unit 110 may update the vehicle path constructed in step 570. Processing unit 110 may update the vehicle path by calculating the distance d k is the distance d i A higher resolution may be used to reconstruct the vehicle path constructed in 570 such that the distance d k may be in the range of 0.1 to 0.3 meters. Processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm, which may result in a cumulative distance vector S corresponding to the total length of the vehicle path (i.e., based on the set of points representing the vehicle path).
[0107] In step 574, the processing unit 110 calculates the look-ahead point ((x l ,z l ) in coordinates. The processing unit 110 may extract look-ahead points from the cumulative distance vector S, and the look-ahead points may be associated with a look-ahead distance and a look-ahead time. The look-ahead distance may have a lower bound range of 10-20 meters and may be calculated as the product of the speed of the vehicle 200 and the look-ahead time. For example, as the speed of the vehicle 200 decreases, the look-ahead distance may also decrease (e.g., until a lower bound is reached). The look-ahead time, which may range from 0.5 to 1.5 seconds, may be inversely proportional to the gain of one or more control loops associated with producing a navigation response in the vehicle 200, such as a heading error tracking control loop. For example, the gain of the heading error tracking control loop may depend on the bandwidth of the yaw rate loop, the steering actuator loop, and the vehicle lateral dynamics, etc. Thus, the higher the gain of the heading error tracking control loop, the shorter the look-ahead time.
[0108] In step 576, processing unit 110 may determine a heading error and yaw rate command based on the look ahead point identified in step 574. Processing unit 110 may calculate the arctangent of the look ahead point, e.g., arctan(x l / z l) to determine the heading error. Processing unit 110 may determine the yaw rate command as the product of the heading error and a high-level control gain. The high-level control gain may be equal to (2 / look ahead time) if the look ahead distance is not at a lower limit. If the look ahead distance is at a lower limit, the high-level control gain may be equal to (2*velocity of vehicle 200 / look ahead distance).
[0109] 5F is a flow chart illustrating an example process 500F for determining whether a leading vehicle is changing lanes, according to a disclosed embodiment. In step 580, processing unit 110 may determine navigation information associated with a leading vehicle (e.g., a vehicle traveling in front of vehicle 200). For example, processing unit 110 may determine a position, a velocity (e.g., direction and speed), and / or an acceleration of the leading vehicle using the techniques described above in connection with FIGs. 5A and 5B. Processing unit 110 may also determine one or more road polynomials, lookahead points (associated with vehicle 200), and / or a snail trail (e.g., a set of points describing a path taken by a leading vehicle) using the techniques described above in connection with FIGs. 5E.
[0110] In step 582, processing unit 110 may analyze the navigation information identified in step 580. In one embodiment, processing unit 110 may calculate a distance (e.g., along the trail) between the snail trail and the road polynomial. If the difference in this distance along the trail exceeds a predetermined threshold (e.g., 0.1-0.2 meters for straight roads, 0.3-0.4 meters for gently curving roads, and 0.5-0.6 meters for sharply curving roads), processing unit 110 may determine that the leading vehicle is likely changing lanes. If multiple vehicles are detected to be traveling in front of vehicle 200, processing unit 110 may compare the snail trails associated with each vehicle. Based on the comparison, processing unit 110 may determine that a vehicle whose snail trail does not match the snail trails of the other vehicles is likely changing lanes. Processing unit 110 may further compare the curvature of the snail trail (associated with the leading vehicle) to the expected curvature of the road segment along which the leading vehicle is traveling. The expected curvature may be extracted from map data (e.g., data from map database 160), road polynomials, snail trails of other vehicles, prior knowledge about the road, etc. If the difference between the snail trail curvature and the expected curvature of the road segment exceeds a predetermined threshold, processing unit 110 may determine that the leading vehicle is likely changing lanes.
[0111] In another embodiment, processing unit 110 may compare the leading vehicle's instantaneous position to the look-ahead point (associated with vehicle 200) over a particular time period (e.g., 0.5-1.5 seconds). If the difference in distance and the cumulative sum of discrepancies between the leading vehicle's instantaneous position and the look-ahead point during the particular time period exceeds a predetermined threshold (e.g., 0.3-0.4 meters for straight roads, 0.7-0.8 meters for gently curving roads, and 1.3-1.7 meters for sharply curving roads), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the snail trail by comparing the lateral distance traveled along the trail to the expected curvature of the snail trail. The expected radius of curvature is calculated as: (δ z 2 +δ x 2 ) / 2 / (δ x ) where σ x represents the lateral movement distance, and σ z represents the longitudinal travel distance. If the difference between the lateral travel distance and the expected curvature exceeds a predetermined threshold (e.g., 500-700 meters), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the position of the leading vehicle. If the position of the leading vehicle obscures the road polynomial (e.g., the leading vehicle overlaps the road polynomial), processing unit 110 may determine that the leading vehicle is likely changing lanes. If the position of the leading vehicle is such that another vehicle is detected ahead of the leading vehicle and the snail trails of the two vehicles are not parallel, processing unit 110 may determine that the (closer) leading vehicle is likely changing lanes.
[0112] In step 584, processing unit 110 may determine whether leading vehicle 200 is changing lanes based on the analyses performed in step 582. For example, processing unit 110 may make that determination based on a weighted average of the individual analyses performed in step 582. Under such a scheme, for example, a determination by processing unit 110 that it is likely that the leading vehicle is changing lanes based on a particular type of analysis may be assigned a value of "1" (with a "0" representing a determination that it is unlikely that the leading vehicle is changing lanes). Different weights may be assigned to different analyses performed in step 582, and the disclosed embodiments are not limited to any particular combination of analyses and weights.
[0113] 6 is a flow chart illustrating an example process 600 for generating one or more navigational responses based on stereo image analysis, according to disclosed embodiments. In step 610, processing unit 110 may receive a first and second plurality of images via data interface 128. For example, a camera included in image acquisition unit 120 (such as image capture devices 122 and 124 having fields of view 202 and 204) may capture and transmit a first and second plurality of images of an area in front of vehicle 200 to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first and second plurality of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.
[0114] In step 620, processing unit 110 may execute stereo image analysis module 404 to perform stereo image analysis of the first and second plurality of images to create a 3D map of the road in front of the vehicle and detect features in the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and road hazards. The stereo image analysis may be performed similar to the steps described above in connection with FIGS. 5A-5D. For example, processing unit 110 may execute stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road marks, traffic lights, road hazards, etc.) in the first and second plurality of images, filter out a subset of the candidate objects based on various criteria, perform a multi-frame analysis, construct measurements, and identify confidence levels for the remaining candidate objects. In performing the above steps, processing unit 110 may consider information from both the first and second plurality of images, rather than information from only one set of images. For example, processing unit 110 may analyze differences in pixel-level data (or other subsets of data from the two streams of captured images) of a candidate object that appears in both the first and second multiple images. As another example, processing unit 110 may estimate the position and / or velocity (e.g., relative to vehicle 200) of a candidate object by observing that an object appears in one of the multiple images but not in the remaining other images, or other differences that may exist for an object that appears in the two image streams. For example, the position, velocity, and / or acceleration relative to vehicle 200 may be determined based on the trajectory, location, movement characteristics, etc. of features associated with the object that appears in one or both of the image streams.
[0115] In step 630, processing unit 110 may execute navigation response module 408 to cause one or more navigational actions in vehicle 200 based on the analysis performed in step 620 and the techniques described above in connection with FIG. 4. The navigational responses may include, for example, turns, lane shifts, acceleration changes, speed changes, braking, etc. In some embodiments, processing unit 110 may use data derived from execution of speed and acceleration module 406 to cause one or more navigational responses. Furthermore, multiple navigational responses may be performed simultaneously, sequentially, or any combination thereof.
[0116] FIG. 7 is a flow chart illustrating an example process 700 for generating one or more navigation responses based on an analysis of three sets of images, according to disclosed embodiments. In step 710, processing unit 110 may receive a first, second, and third plurality of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture and transmit a first, second, and third plurality of images of an area in front of and / or to the side of vehicle 200 to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first, second, and third plurality of images via three or more data interfaces. For example, each of image capture devices 122, 124, and 126 may have an associated data interface that communicates data to processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.
[0117] In step 720, processing unit 110 may analyze the first, second, and third pluralities of images to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and road hazards. The analysis may be performed similar to the steps described above in connection with FIGS. 5A-5D and 6. For example, processing unit 110 may perform monocular image analysis on each of the first, second, and third pluralities of images (e.g., via execution of monocular image analysis module 402 and based on the steps described above in connection with FIGS. 5A-5D). Alternatively, processing unit 110 may perform stereo image analysis on the first and second pluralities of images, the second and third pluralities of images, and / or the first and third pluralities of images (e.g., via execution of stereo image analysis module 404 and based on the steps described above in connection with FIG. 6). Processed information corresponding to the analysis of the first, second, and / or third pluralities of images may be combined. In some embodiments, processing unit 110 may perform a combination of monocular image analysis and stereo image analysis. For example, processing unit 110 may perform monocular image analysis on the first plurality of images (e.g., via execution of monocular image analysis module 402) and stereo image analysis on the second and third plurality of images (e.g., via execution of stereo image analysis module 404). The configuration of image capture devices 122, 124, and 126 (including their respective positions and fields of view 202, 204, and 206) may affect the type of analysis performed on the first, second, and third plurality of images. The disclosed embodiments are not limited to a particular configuration of image capture devices 122, 124, and 126 or the type of analysis performed on the first, second, and third plurality of images.
[0118] In some embodiments, processing unit 110 may run tests on system 100 based on the images acquired and analyzed in steps 710 and 720. Such tests may provide an indicator of the overall performance of system 100 with a particular configuration of image capture devices 122, 124, and 126. For example, processing unit 110 may identify the rate of "false hits" (e.g., when system 100 incorrectly determines that a vehicle or pedestrian is present) and "misses."
[0119] At step 730, processing unit 110 may cause one or more navigation responses in vehicle 200 based on information derived from two of the first, second, and third pluralities of images. The selection of two of the first, second, and third pluralities of images may depend on various factors, such as, for example, the number, type, and size of objects detected in each of the multiple images. Processing unit 110 may make the selection based on the quality and resolution of the images, the effective field of view reflected in the images, the number of captured frames, the degree to which one or more objects of interest actually appear in the frames (e.g., the percentage of frames in which the object appears, the proportion in which the object appears in each such frame, etc.), etc.
[0120] In some embodiments, processing unit 110 may select information derived from two of the first, second, and third plurality of images by identifying the extent to which information derived from one image source is consistent with information derived from the other image source. For example, processing unit 110 may combine processed information derived from each of image capture devices 122, 124, and 126 (whether monocular analysis, stereo analysis, or any combination of the two) to identify visual indicators (e.g., lane markings, detected vehicles and / or their positions and / or paths, detected traffic lights, etc.) that are consistent across images captured from each of image capture devices 122, 124, and 126. Processing unit 110 may also filter out information that is inconsistent across captured images (e.g., vehicles changing lanes, lane models showing vehicles too close to vehicle 200, etc.). Thus, processing unit 110 may select information derived from two of the first, second, and third plurality of images based on identifying consistent and inconsistent information.
[0121] The navigational responses may include, for example, turns, lane shifts, and acceleration changes. Processing unit 110 may generate one or more navigational responses based on the analysis performed in step 720 and the techniques described above in connection with FIG. 4. Processing unit 110 may also generate one or more navigational responses using data derived from execution of velocity and acceleration module 406. In some embodiments, processing unit 110 may generate one or more navigational responses based on a relative position, relative velocity, and / or relative acceleration between vehicle 200 and an object detected in any of the first, second, and third multiple images. The multiple navigational responses may be performed simultaneously, sequentially, or any combination thereof.
[0122] Prediction of Cut-In Vehicles and Altruistic Behavioral Responses During navigation, an autonomous vehicle such as vehicle 200 may encounter other vehicles attempting to lane shift. For example, a vehicle in a lane to the left or right of the lane in which vehicle 200 is traveling (e.g., a lane designated by markings on the road or a lane unmarked on the road that is aligned with the path of vehicle 200) may attempt to change or cut into the lane in which vehicle 200 is traveling. Such a vehicle may be a target vehicle. When such a cut-in occurs, vehicle 200 may need to make a navigation response. For example, vehicle 200 may change its speed or acceleration and / or change into another lane to avoid being cut in by the target vehicle.
[0123] In some instances, the target vehicle may appear to be about to cut in but ultimately does not. The driver of the target vehicle (or even a fully or partially autonomous navigation system associated with the target vehicle) may, for example, have a change of mind or otherwise change the navigation plan to something other than a lane change, or the target vehicle may simply skid. Thus, to avoid frequent unnecessary braking and / or acceleration, it may be desirable for the vehicle 200 to delay performing a navigation response until it is determined that a cut-in by the target vehicle is likely enough. On the other hand, in some situations (especially when the target vehicle is expected to change course to the host vehicle's path), it may be desirable for the vehicle 200 to perform a navigation response earlier. Such navigation based at least in part on expected behavior may help to avoid hard braking and may provide a much larger safety margin. Improving prediction of cut-in attempts by the target vehicle may help to minimize both unnecessary braking and hard braking. Such improved prediction may be referred to as cut-in detection, and the navigation response taken when a cut-in is detected may be referred to as a cut-in response.
[0124] In some embodiments, such improved predictions may rely on, for example, monocular and / or stereoscopic image analysis, and / or information obtained from other sources (e.g., from GPS devices, speed sensors, accelerometers, suspension sensors, etc.) to detect static road features (e.g., lane endings, road divisions), dynamic road features (e.g., the presence of other vehicles ahead of the vehicle that may be attempting to cut in), and / or traffic rules and driving habits in a geographic region. These static road features, dynamic road features, and / or traffic rules and driving habits may be referred to as predefined cut-in sensitivity modifiers, and the presence of one or more predefined cut-in sensitivity modifiers may cause vehicle 200 to change its sensitivity to cut-in attempts by a target vehicle. For example, if a predefined cut-in sensitivity modifier is present in the environment (e.g., the target vehicle is closely following another vehicle traveling at a slower speed), vehicle 200 may rely on a first cut-in sensitivity parameter, and if no predefined cut-in sensitivity modifier is present in the environment (e.g., there is only the target vehicle in the lane), vehicle 200 may rely on a second cut-in sensitivity parameter. The second interruption sensitivity parameter may be different (eg, more sensitive) than the first interruption sensitivity parameter.
[0125] In some cases, it may be necessary for the target vehicle to cut in. For example, the lane in which the target vehicle is traveling may end, the road may split, or there may be an obstacle (e.g., a stopped vehicle, an object, or other type of obstacle) in the lane in which the target vehicle is traveling. However, in other cases, cutting in by the target vehicle may be optional. For example, the target vehicle may simply attempt to cut in to pass a vehicle moving at a slower speed. If cutting in is optional, whether the target vehicle attempts to cut in may depend on how the vehicle 200 (e.g., the host vehicle) behaves. For example, the vehicle 200 may signal that the target vehicle may cut in by slowing down, or may signal that the target vehicle should not cut in by accelerating. Although accelerating may often be desirable for the vehicle 200 as long as it enables the vehicle 200 to reach its destination sooner, the operator of the vehicle 200 (or a navigation system that fully or partially controls the vehicle 200) may be inclined to allow the cut in in some or all cases. This may be referred to as altruistic behavior. Thus, it may be desirable for vehicle 200 to take into account considerations regarding altruistic behavior when determining whether to allow a target vehicle to cut in place.
[0126] 8 is an example functional block of memory 140 and / or 150 that may be stored / programmed with instructions to perform one or more operations according to the disclosed embodiments. Although the following is directed to memory 140, one skilled in the art will recognize that instructions may be stored in memory 140 and / or 150 and that the instructions may be executed by processing unit 110 of system 100.
[0127] As shown in FIG. 8, memory 140 may include monocular image analysis module 402, stereo image analysis module 404, velocity and acceleration module 406, and navigation response module 408, which may take any of the forms of modules described above in connection with FIG. 4. Memory 140 may further include interrupt detection module 802, interrupt response module 804, and altruistic behavior module 806. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 402-408 and 802-806 included in memory 140. Those skilled in the art will appreciate that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Thus, any steps of the following processes may be performed by one or more processing devices. Further, any of modules 402-408 and 802-806 may be stored remotely from vehicle 200 (e.g., distributed across one or more servers connected to a network and accessible over the network via wireless transceiver 172 of vehicle 200).
[0128] In some embodiments, the cut-in detection module 802 may store instructions that, when executed by the processing unit 110, enable detection of a target vehicle. The target vehicle may be a vehicle traveling in a lane adjacent to the lane in which the vehicle 200 is traveling, and in some cases may be a leading vehicle. In some embodiments, the cut-in detection module 802 may detect the target vehicle by performing monocular image analysis of a set of images acquired by one of the image capture devices 122, 124, and 126, as described above in connection with the monocular image analysis module 402. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensor information (e.g., information from a radar or lidar) to perform the monocular image analysis. As described above in connection with FIGS. 5A-5D, such monocular image analysis may include detecting a set of features in the set of images, such as vehicle edge features, vehicle lights (or other elements associated with the vehicle), lane markings, vehicles, or road signs. For example, detecting a target vehicle may be performed as described in connection with Figure 5B, including identifying a set of candidate objects including the target vehicle, filtering the set of candidate objects, performing a multi-frame analysis of the set of candidate objects, constructing a set of measurements of the detected objects (including the target vehicle), and performing an optical flow analysis. The measurements may include, for example, position, velocity, and acceleration values (for vehicle 200) associated with target coordinates.
[0129] Alternatively or additionally, in some embodiments, cut-in detection module 802 may detect the target vehicle by performing stereo image analysis of the first and second sets of images acquired by a combination of image capture devices selected from image capture devices 122, 124, and 126, as described above in connection with stereo image analysis module 404. In some embodiments, processing unit 110 may combine information from the first and second sets of images with additional sensor information (e.g., information from radar or lidar) to perform stereo image analysis. For example, stereo image analysis module 404 may include instructions to perform stereo image analysis based on the first set of images acquired by image capture device 124 and the second set of images acquired by image capture device 126. As described in connection with FIG. 6, stereo image analysis may include detecting a set of features in the first and second sets of images, such as vehicle edge features, vehicle lights (or other elements associated with the vehicle), lane markings, cars, or road signs.
[0130] In other embodiments, as an alternative to detecting one or more vehicles (e.g., target vehicles) by analyzing images acquired by one of image capture devices 122, 124, and 126, cut-in detection module 802 may instead detect vehicles through analysis of sensor information, such as information acquired via a radar or lidar device included in system 100.
[0131] In some embodiments, the cut-in detection module 802 may further store instructions that, when executed by the processing unit 110, enable identification of indicators that the target vehicle will attempt to cut-in (i.e., the target vehicle will attempt to change lanes into the lane in which the vehicle 200 is traveling or otherwise move into the host vehicle's travel path). In some embodiments, identifying the indicators may include detecting the position and / or speed of the target vehicle using monocular and / or stereo image analysis, as described above in connection with FIG. 5B. In some embodiments, identifying the indicators may further include detecting one or more road markings, as described above in connection with FIG. 5C. Also, in some embodiments, identifying the indicators may further include detecting that the target vehicle is changing lanes, as described above in connection with FIG. 5F. In particular, the processing unit 110 may use the techniques described above in connection with FIGS. 5A and 5B to identify navigation information associated with the target vehicle, such as the position, speed (e.g., direction and speed), and / or acceleration of the leading vehicle. Processing unit 110 may also identify one or more road polynomials, look-ahead points (associated with vehicle 200), and / or snail trails (e.g., a set of points describing a path taken by a leading vehicle) using the techniques described above in connection with FIG. 5E. Additionally, processing unit 110 may analyze the navigation information, for example, by calculating a distance (e.g., along the trail) between the snail trail and the road polynomials, or by comparing the instantaneous position of the leading vehicle to the look-ahead points (associated with vehicle 200) over a particular period of time (e.g., 0.5-1.5 seconds), as described above in connection with FIG. 5F. Based on the analysis, processing unit 110 may identify whether an indicator is present that the leading vehicle is attempting to cut in.
[0132] The cut-in detection module 802 may further store instructions that, when executed by the processing unit 110, enable detection of the presence of a predefined cut-in sensitivity modifier in the environment of the vehicle 200. The predefined cut-in sensitivity modifier may include any indicator that suggests a tendency of the target vehicle to remain on its current path or to change course into the path of the host vehicle. Such sensitivity modifiers may include static road features (e.g., lane endings, road divisions, barriers, objects), dynamic road features (e.g., the presence of other vehicles ahead of the vehicle that may be attempting to cut-in), and / or traffic rules and driving habits in a geographical area. In some embodiments, detecting the predefined cut-in sensitivity modifier may include using monocular and / or stereo image analysis to detect a set of features in a set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazards, and any other features associated with the vehicle's environment, such as those described above in connection with FIGS. 5A-5D and 6. Alternatively or additionally, in some embodiments, detecting the predetermined cut-in sensitivity modifier may include using other sensor information, such as GPS data. For example, lane endings, highway divisions, etc. may be identified based on map data. As another example, traffic rules may be identified based on GPS location data.
[0133] The interrupt response module 804 may store instructions that, when executed by the processing unit 110, enable an interrupt response to be performed. The interrupt response may take any of the forms described above with respect to the navigation response, such as a turn, a lane shift, an acceleration change, etc., which are discussed later in connection with the navigation response module 408.
[0134] In some embodiments, whether or not a cut-in response is performed may depend on the presence or absence of a predefined cut-in sensitivity modifier in the environment. For example, if a predefined cut-in sensitivity modifier is not present in the environment (e.g., there are only vehicles attempting to cut-in in the lane, no lane shifts or obstacles are detected either through image analysis or review of GPS / map / traffic data, etc.), vehicle 200 may rely on the first cut-in sensitivity parameter. In some embodiments, if a predefined cut-in sensitivity modifier is not present in the environment, vehicle 200 may rely on a value associated with the first cut-in sensitivity parameter. The value may include any value directly tied and / or measured to the first cut-in sensitivity parameter, or may be one or more values indirectly related to the first cut-in sensitivity parameter.
[0135] On the other hand, if one or more predefined lurch sensitivity modifiers are present in the environment (e.g., the target vehicle is moving at a high speed and is approaching a slower vehicle in its lane, a lane shift or lane ending condition is detected visually through image analysis or through GPS / map / traffic reference, the local area is an area that discourages left lane driving, etc.), the vehicle 200 may rely on a second lurch sensitivity parameter. The second lurch sensitivity parameter may be different (e.g., more sensitive) than the first lurch sensitivity parameter. In some embodiments, if one or more predefined lurch sensitivity modifiers are present in the environment, the vehicle 200 may rely on a value associated with the second lurch sensitivity parameter. Similar to the above discussion regarding the value associated with the first lurch sensitivity parameter, the value associated with the second lurch sensitivity parameter may include any value directly tied and / or measured for the second lurch sensitivity parameter, or may be one or more values indirectly related to the second lurch sensitivity parameter.
[0136] The first and second interruption sensitivity parameters may establish two operating states, a first in which the host vehicle may be less sensitive to movements or course changes by the target vehicle, and a second in which the host vehicle may be more sensitive to movements or course changes by the target vehicle. Such a state-based approach may reduce the number or extent of decelerations, accelerations, or course changes in the first state if course changes by the target vehicle are not expected based on any environmental conditions. However, in the second state, if at least one condition is detected or identified in the target vehicle's environment that is expected to cause the target vehicle to change course (or other navigation response), the host vehicle may be more sensitive to the target vehicle's course or navigation changes and may change course, accelerate, decelerate, etc. based on even slight changes recognized in the target vehicle's navigation. Such small changes in the second state may be interpreted as consistent with expected navigation changes by the target vehicle, and therefore may ensure a more rapid response compared to navigation in the first state.
[0137] Further, in the second state, the host vehicle's course change may be performed based solely on detection of the sensitivity modifier, without detecting a navigation response or change of the target vehicle. For example, if the host vehicle system's image analysis indicates the presence of a sensitivity modifier (e.g., by recognizing a road sign indicating a lane end condition in the captured image, recognizing a lane end ahead of the target vehicle in the image, recognizing an obstacle or a slower moving vehicle ahead of the target vehicle, or any other sensitivity modifier), the host vehicle may take one or more preemptive actions (slow down, accelerate, change course, etc.) even without detecting a navigation change by the target vehicle. Such changes may be warranted based on an expectation that the target vehicle will need to make a navigation change in the future. Such changes may increase or maximize the amount of time between the host vehicle's navigation change and the expected target vehicle's navigation change.
[0138] The cut-in sensitivity parameter, as described above with respect to FIG. 5F, may take the form of a threshold value (e.g., a distance threshold value between the snail trail and the road polynomial (e.g., along the trail), or a distance threshold value between the target vehicle's instantaneous position and a look-ahead point (associated with vehicle 200) for a particular time period (e.g., 0.5-1.5 seconds)), a combination of such threshold values (e.g., a combination of a target vehicle's lateral movement speed threshold value and a vehicle's lateral position threshold value relative to vehicle 200), and / or a weighted average of such threshold values). As previously described, in some embodiments, the cut-in sensitivity parameter may also be any arbitrarily assigned variable whose value is associated with the parameter, or any value indirectly related to the value of the parameter may be used to set the two (or more) sensitivity states described above. For example, such a variable may be associated with a first value indicative of a low sensitivity state when no sensitivity modifying element is detected, and associated with a second value indicative of a higher sensitivity state when a sensitivity modifying element is found in the target vehicle's environment.
[0139] In some embodiments, the interruption response module 804 may evaluate the distance from the vehicle 200 to the vehicle following the vehicle 200 and include the distance in the interruption sensitivity parameter. For example, the vehicle 200 may include one or more rear sensors (e.g., radar sensors) to determine the distance from the vehicle 200 to the vehicle following the vehicle 200. Furthermore, in some embodiments, the vehicle 200 may include one or more rear cameras that provide images to the system 100 for analysis to determine the distance from the vehicle 200 to the vehicle following the vehicle 200. In yet another embodiment, the system 100 may use data from the one or more sensors and from the one or more rear cameras to determine the distance from the vehicle 200 to the vehicle following the vehicle 200. Based on the distance from the vehicle 200, the interruption response module 804 may evaluate whether it is safe for the vehicle 200 to slow down based on the distance between the vehicle 200 and the vehicle following the vehicle 200.
[0140] Alternatively or additionally, the cut-in sensitivity parameter may be derived from examples using machine learning techniques such as neural networks. For example, the neural network may be trained to identify the cut-in sensitivity parameter based on a scenario. As an example, the neural network may be trained for a scenario in which a road includes three driving lanes, the vehicle 200 is driving in a center lane, there is a leading vehicle in the center lane, two vehicles are in the left lane, and one vehicle is in the right lane. The scenario may specify to the neural network the longitudinal position, longitudinal velocity, lateral position, and lateral velocity of each of the vehicles. Based on the scenario, the neural network may identify, for example, a binary output (e.g., whether the target vehicle will attempt to cut-in within the next N frames) or a value output (e.g., the time until the target vehicle attempts to cut-in to the center lane). The binary or value output, or other values derived from these outputs, may be used as the cut-in sensitivity parameter. The neural network may be trained for other scenarios as well, such as a scenario in which the road has only two driving lanes, there is no leading vehicle in the center lane, or there is no vehicle in the right lane. In some embodiments, the neural network may be provided via one or more programming modules stored in memory 140 and / or 150. In another embodiment, in addition to or as an alternative to memory 140 and / or 150, the neural network (or aspects of the neural network) may be provided via one or more servers located remotely from vehicle 200 and accessible over a network via wireless transceiver 172.
[0141] In some embodiments, to minimize the number of scenarios, the scenarios may be classified according to the number of lanes on the road and the location of the vehicle 200. For example, the scenarios may include a scenario with two lanes and the vehicle 200 in the left lane, a scenario with two lanes and the vehicle 200 in the right lane, a scenario with three lanes and the vehicle 200 in the left lane, a scenario with three lanes and the vehicle 200 in the center lane, a scenario with three lanes and the vehicle in the right lane, a scenario with four lanes and the vehicle 200 in the leftmost lane, a scenario with four lanes and the vehicle 200 in the center left lane, etc. In each scenario, a leading vehicle may be traveling in the same lane as the vehicle 200, and two vehicles may be traveling in each of the other lanes. To account for scenarios where one or more of these other vehicles are absent (e.g., no leading vehicle, only one vehicle in the right lane, etc.), the longitudinal distance of the absent vehicles may be set to infinity.
[0142] As described above, once vehicle 200 detects a target vehicle (e.g., identifies an indicator that the target vehicle is attempting to cut in) and detects whether any sensitivity modifiers are present in the environment, cut-in response module 804 may provide the target vehicle, indicators, and / or sensitivity modifiers (if present) to a neural network, which may select a scenario that most closely resembles the situation of vehicle 200. Based on the selected scenario, the neural network may indicate a binary output (e.g., whether the target vehicle will attempt to cut in during the next N frames) or a value output (e.g., the time until the target vehicle attempts to cut in to the center lane), as described above, from which a cut-in sensitivity parameter may be derived. Based on the cut-in sensitivity parameter, cut-in response module 804 causes a cut-in response to be performed.
[0143] As mentioned above, the interrupt response may take any of the forms described above in connection with the navigation response, such as a turn, lane shift, and / or change in acceleration, as discussed above with respect to the navigation response module 408. In some embodiments, the processing unit 110 may use data derived from execution of the speed and acceleration module 406 described above to generate one or more interrupt responses. Additionally, multiple interrupt responses may occur simultaneously, sequentially, or any combination thereof. For example, the processing unit 110 may cause the vehicle 200 to shift one lane and then accelerate, e.g., by sequentially sending control signals to the steering system 240 and the throttle system 220 of the vehicle 200. Alternatively, the processing unit 110 may cause the vehicle 200 to brake simultaneously while shifting lanes, e.g., by simultaneously sending control signals to the braking system 230 and the steering system 240 of the vehicle 200. In some embodiments, interruption response module 804 may analyze images acquired by one of image capture devices 122, 124, and 126 to detect whether any interruption sensitivity modifiers are present in the environment of vehicle 200. In other embodiments, in addition to or as an alternative to analyzing images, interruption response module 804 may detect whether any interruption sensitivity modifiers are present in the environment of vehicle 200 through analysis of sensor information, such as information acquired through a radar or lidar device included in system 100.
[0144] In some embodiments, the neural network may be further configured to determine an interruption response. For example, the neural network may determine and output an indication of which navigation response to perform based on the detected target vehicle, indicators, and / or sensitivity modifiers (if any). In some embodiments, such output may be determined based on previously detected driver behavior. In some embodiments, previously detected driver behavior may be filtered using a cost function analysis that prioritizes smooth driving behavior (e.g., smooth driving behavior may be measured as minimizing the square of the integral of deceleration or acceleration over a period of time).
[0145] The interrupt detection module 802 and the interrupt response module 804 are further described below in conjunction with FIGS.
[0146] Altruistic behavior module 806 may store instructions that, when executed by processing unit 110, take into account considerations related to altruistic behavior to determine whether vehicle 200 should allow a target vehicle to cut into the lane in which vehicle 200 is traveling. To this end, processing unit 110 may detect the target vehicle by any of the methods described above in connection with cut-in detection module 802.
[0147] Additionally, processing unit 110 may identify one or more situational features associated with the target vehicle. The situational feature may be, for example, any feature that indicates that the target vehicle would benefit from changing lanes into the lane in which vehicle 200 is traveling. For example, the situational feature may indicate that the target vehicle is traveling in a lane adjacent to the lane in which vehicle 200 is traveling, and that the target vehicle is behind another vehicle traveling slower than the target vehicle. In general, the situational feature may indicate that a cut-in or other navigational response by the target vehicle may not be necessary, but that a cut-in would be advantageous for the target vehicle. Other such situations may include, for example, during heavy traffic, congested detours, end-of-lane situations, or any other situation in which the target vehicle may wish to move into the path of the host vehicle.
[0148] In some embodiments, detecting the situational feature may include detecting a set of features in the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazards, and any other features associated with the vehicle's environment, using monocular and / or stereo image analysis, as described above in connection with FIGS. 5A-5D and 6. For example, identifying the situational feature may include detecting a position and / or speed of the target vehicle and / or one or more other vehicles, using monocular and / or stereo image analysis, as described above in connection with FIG. 5B. In some embodiments, identifying the situational feature may further include detecting one or more road markings, as described above in connection with FIG. 5C. Alternatively or additionally, in some embodiments, detecting the situational feature may include using other sensor information, such as GPS data. For example, roadway on-ramps may be identified based on map data. In some situations, detecting the situational feature may include identifying, from the captured images, the position of the target vehicle relative to other vehicles, the number of vehicles in the vicinity of the target vehicle, and the like. In yet other embodiments, detecting the situational characteristic may include using other sensor information, such as information obtained via a radar or lidar device included in system 100.
[0149] The altruistic behavior module 806 may further store instructions that, when executed by the processing unit 110, determine whether to cause a navigation change (e.g., any of the navigation responses described above) to permit a target vehicle to cut into the lane in which the vehicle 200 is traveling. Such a determination may be made based on the altruistic behavior parameters.
[0150] The altruistic behavior parameter may be set, for example, based on input from an operator of the vehicle 200. For example, the operator may set the altruistic behavior to allow all target vehicles to cut into the lane in which the vehicle 200 is moving, to allow only one out of every n target vehicles to cut into the lane in which the vehicle 200 is moving, to allow a target vehicle to cut into the lane in which the vehicle 200 is moving only if other vehicles ahead of the target vehicle are traveling below a certain speed, etc. Alternatively or additionally, in some embodiments, the altruistic behavior parameter may be user selectable (e.g., a user selectable value or state) before or during navigation. For example, the user may select (e.g., through the user interface 170) whether to be altruistic during navigation when entering the vehicle 200. As another example, the user may select (e.g., through the user interface 170) whether to be altruistic in that case when one or more situational features are detected.
[0151] In some embodiments, the altruistic behavior parameter may be set based on at least one information element determined by analyzing calendar entry information for an operator or passenger of the vehicle 200. For example, in some embodiments, the altruistic behavior parameter may indicate that a target vehicle is permitted to cut into the lane in which the vehicle 200 is traveling as long as the vehicle 200 arrives at the destination within a time frame acceptable to the operator of the vehicle 200. For example, if the operator or passenger is not in a hurry, the criteria may allow more target vehicles to cut in, but if the operator or passenger is in a hurry, the criteria may allow fewer target vehicles to cut in, or may not allow any target vehicles to cut in at all. Whether the operator or passenger is in a hurry may be determined by the processing unit 110 based on, for example, calendar entry information associated with the operator or passenger (e.g., a calendar event indicating that a user wants to arrive at a particular location by a particular time) and navigation data for the vehicle 200 (e.g., GPS and / or map data predicting the arrival time at the location).
[0152] In some embodiments, the altruistic behavior parameter may be set based on an output of a random number generation function. For example, a particular output of the random number generation function may cause the vehicle 200 to allow the target vehicle 1002 to cut into the first lane 1004, while another output of the random number generation function may not cause the vehicle 200 to allow the target vehicle 1002 to cut into the first lane 1004. Further alternatively or additionally, the altruistic behavior parameter may be set based on a predetermined number of encounters with a target vehicle where one or more situational characteristics indicate that the target vehicle would benefit from a course change into the first lane 1004.
[0153] In some embodiments, the altruistic behavior parameter may be constant. Alternatively, the altruistic behavior parameter may be updated such that a navigational change of vehicle 200 occurs at least a predetermined percentage of encounters with a target vehicle where one or more situational characteristics indicate that the target vehicle would benefit from a course change to the path that vehicle 200 is on. For example, the predetermined percentage may be at least 10%, at least 20%, at least 30%, etc.
[0154] In some embodiments, the altruistic behavior parameters may specify rules such that navigation changes are implemented when certain criteria are met. For example, a given altruistic behavior parameter may indicate that if vehicle 200 is approaching a target vehicle in an adjacent lane and the target vehicle is traveling behind another vehicle traveling slower than the target vehicle, vehicle 200 should slow down to allow the target vehicle to cut in, unless the target vehicle indicates a desire to cut in (e.g., through use of a turn signal or through a lateral movement toward the lane in which vehicle 200 is traveling), vehicle 200 is slowing down below a certain threshold (e.g., to avoid excessive hard braking), and there is no vehicle behind vehicle 200 that would prevent vehicle 200 from safely slowing down.
[0155] In some embodiments, the altruistic behavior parameter may be designed to be inconsistent, such that the same situational features and criteria may result in different navigational changes or no navigational changes at all. For example, the altruistic behavior parameter may result in various random and / or periodic altruistic actions. As one example, the altruistic behavior parameter may allow only one out of every n target vehicles to cut in. In some embodiments, n may be randomly selected, may vary randomly, may increase each time a situational feature is detected, and / or may be reset to a lower value once cutting in is allowed. As another example, the altruistic behavior parameter may determine whether to allow a target vehicle to cut in based on a comparison of a randomly generated number to a predetermined threshold.
[0156] In some embodiments, the altruistic behavior module 804 may consider how many vehicles are traveling behind the vehicle 200 and add this information to the altruistic behavior parameters. For example, the vehicle 200 may include one or more rear sensors (e.g., radar sensors) to detect following vehicles and / or one or more rear cameras that provide images and / or wireless traffic information to the system 100 for analysis. In this case, based on the number of vehicles following the vehicle 200, the altruistic behavior module 804 may evaluate the potential impact of whether the vehicle 200 allows the target vehicle to cut in. For example, if it is determined that a long line of vehicles (e.g., 5, 10, or more vehicles) is traveling behind the vehicle 200, if the vehicle 200 does not allow the target vehicle to cut in, the target vehicle is likely unlikely to have an opportunity to cut in until the vehicles behind have passed, which may take a significant amount of time. However, if only a small number of vehicles (e.g., one or two) are traveling behind vehicle 200, the target vehicle will have a short opportunity to cut in (e.g., after the small number of vehicles have passed) even if vehicle 200 does not allow the target vehicle to cut in.
[0157] In some embodiments, the altruistic behavior parameter may be derived from examples using machine learning techniques such as neural networks. For example, a neural network may be trained to identify the altruistic behavior parameter based on a scenario, as described above. In some embodiments, the altruistic behavior parameter may be identified based on previously detected driver behavior. For example, altruistic behavior by the driver may be weighted positively, such that the neural network prioritizes altruistic behavior. As described above, inconsistency may be added through random or periodic variation, as described above.
[0158] The altruistic behavior module 806 is further described below with respect to FIGS.
[0159] FIG. 9A is a diagram of an exemplary situation in which vehicle 200 may detect and respond to a cut-in. As shown, vehicle 200 may be traveling alongside target vehicle 902 on road 900. Target vehicle 902 may be traveling in a first lane 904, and vehicle 200 may be traveling in a second lane 906. Although first lane 904 is shown to be the left lane and second lane 906 is shown to be the right lane, first and second lanes 904 and 906 may be any adjacent lanes on road 900. Additionally, while only two lanes 904, 906 are shown on road 900, it should be understood that there may be more lanes. Additionally, the term "lane" may be understood to refer more generally to a path along which vehicle 200 and target vehicle 902 are traveling. For example, in some embodiments, reference to "lane" may refer to a path aligned with the direction of travel or path of vehicle 200.
[0160] Vehicle 200 may be configured to receive images of the environment surrounding road 900 from one or more image capture devices associated with vehicle 200, such as, for example, image capture devices 122, 124, and / or 126. Vehicle 200 may also receive other sensor information, such as GPS data, map data, radar data, or lidar data. Based on the images and / or other sensor information, vehicle 200 may detect target vehicle 902. For example, vehicle 200 may identify an image of target vehicle 902 in a plurality of images. Vehicle 200 may detect target vehicle 902 in other ways as well, including any of the methods described above in connection with cut-in detection module 802.
[0161] Further based on the image and / or other sensor information, the vehicle 200 may identify at least one indicator that the target vehicle 902 will change from the first lane 904 to the second lane 906. For example, the vehicle 200 may detect the position and / or speed of the target vehicle 902 as described in connection with FIG. 5B and / or the position of one or more road marks on the road 900 as described in connection with FIG. 5C based on monocular and / or stereo image analysis of the image (or based on other sensor data, such as radar or lidar data). As another example, the vehicle 200 may detect that the target vehicle 902 has started to change lanes as described above in connection with FIG. 5F. As yet another example, the vehicle 200 may detect that the first lane 904 is ending based on map data. The vehicle 200 may identify the at least one indicator in other ways as well, including any of the ways described above in connection with the cut-in detection module 802.
[0162] As previously discussed, when an indicator is detected, vehicle 200 may determine whether to take a navigational response. To minimize both unnecessary braking and sudden braking, vehicle 200 may consider the presence or absence of a predefined cutting-in sensitivity modifier affecting the likelihood that target vehicle 902 will cut-into second lane 906 in making this determination. If the predefined cutting-in sensitivity modifier is not detected, a navigational response may be generated in vehicle 200 based on the identification of the indicator and based on a first cutting-in sensitivity parameter. On the other hand, if the predefined sensitivity factor is detected, a navigational response may be generated in vehicle 200 based on the identification of the indicator and based on a second cutting-in sensitivity parameter. The second cutting-in sensitivity parameter may be different (e.g., more sensitive) than the first cutting-in sensitivity parameter.
[0163] Additional example situations involving predetermined interrupt sensitivity modifiers are illustrated in Figures 9B-9E.
[0164] 9B illustrates an example predefined cutting-in sensitivity modifier, which takes the form of an obstacle in a first lane 904. As shown, a target vehicle 902 is traveling in the first lane 904 and vehicle 200 is traveling in a second lane 906. The obstacle may act as a predefined cutting-in sensitivity modifier if it makes the target vehicle 200 more likely to attempt a cutting-in than would be the case in the absence of the obstacle.
[0165] As shown, an obstacle may be detected through detection of another vehicle 908 traveling ahead of the target vehicle 902 in the first lane 904. While the obstacle is shown as another vehicle 908, in some embodiments the obstacle may take the form of a stopped vehicle, an accident, a hazard, a pedestrian, etc. In one embodiment where the obstacle is another vehicle 908, the vehicle 200 may detect the obstacle by detecting that the other vehicle 908 is traveling slower than the target vehicle 902. This is because if the other vehicle 908 is traveling slower than the target vehicle 902, the slower speed of the other vehicle 908 makes it more likely that the target vehicle 902 will attempt to cut into the second lane 906. Such obstacles may constitute a predetermined cut-in sensitivity modifier.
[0166] Vehicle 200 may detect obstacles in any of the ways described above for detecting sensitivity modifying elements in connection with FIG. 8. For example, vehicle 200 may detect the position and / or speed of other vehicles 908 in addition to those of target vehicle 902 based on monocular and / or stereo image analysis of images. The position and / or speed of other vehicles 908 may also be detected based on other sensor information. As another example, vehicle 200 may detect obstacles (e.g., other vehicles 908) based on GPS data, map data, and / or traffic data from a traffic application such as Waze. As yet another example, vehicle 200 may detect obstacles through analysis of radar or lidar data.
[0167] When the predetermined cutting-in sensitivity modifier is detected (i.e., when the vehicle 200 detects that an obstacle is on the first lane 904), the vehicle 200 may cause a navigation response based on the identification of the indicator and based on the second cutting-in sensitivity parameter. The navigation response may include, for example, accelerating the vehicle 200, decelerating the vehicle 200, or (if possible) changing lanes of the vehicle 200. The second cutting-in sensitivity parameter may be more sensitive than the first cutting-in sensitivity parameter in the absence of the predetermined cutting-in sensitivity modifier because the presence of the predetermined cutting-in sensitivity modifier increases the likelihood that the target vehicle 902 will attempt to cut-in to the second lane 906.
[0168] In some embodiments, the second cut-in sensitivity parameter may depend on the presence and actions of vehicles surrounding the vehicle 200. For example, the second cut-in sensitivity parameter may take into account the lateral speed and lateral position of the target vehicle 902, and the sensitivity of the second cut-in sensitivity parameter may be correlated to respective thresholds for the lateral speed and lateral position of the target vehicle 902. For example, a high lateral speed threshold and / or a low lateral position threshold may result in less sensitivity, resulting in a slower navigation response, as compared to a low lateral speed threshold and / or a high lateral position threshold, which may result in higher sensitivity and a quicker navigation response.
[0169] Higher sensitivity (i.e., a more sensitive cut-in sensitivity parameter) may be desirable in certain circumstances. For example, if the target vehicle 902 is moving faster than the vehicle 200 and the other vehicle 908 is moving much slower than the target vehicle 902 in the first lane 904, it may appear that the target vehicle 902 may be forced to modify its behavior by either slowing down and staying in the first lane 904, slowing down and changing into the second lane 906 behind the vehicle 200, changing into another lane to the left of the first lane 904 (if such a lane exists), or cutting into the second lane 906. If the target vehicle 902 must decelerate rapidly to avoid a collision with the other vehicle 908, cutting into the second lane 906 becomes more likely. Similarly, if the target vehicle 902 must decelerate rapidly to change into the second lane 906 behind the vehicle 200, cutting into the second lane 906 becomes more likely.
[0170] The sensitivity of the second cut-in sensitivity parameter may further depend on the presence and actions of other vehicles around vehicle 200. For example, the sensitivity may be lower if there are no other vehicles 908 in the first lane 904 and the distance between vehicle 200 and the closest vehicle preceding vehicle 200 in the second lane 906 is short. As another example, the sensitivity may be lower if other vehicles 908 in the first lane 904 are traveling at approximately the same speed and / or faster than the target vehicle 902 and the distance between vehicle 200 and the closest vehicle preceding vehicle 200 in the second lane 906 is short. As yet another example, if the target vehicle 902 is in the passing lane (e.g., if the first lane 904 is to the left of the second lane 906 in a country where vehicles drive on the right side of the road 900), another vehicle 908 in the first lane 904 is moving at approximately the same speed and / or faster than the target vehicle 902, and there is a long distance between the second vehicle 200 in the second lane 906 and the closest vehicle preceding the vehicle 200, the sensitivity may be slightly higher (e.g., low to medium sensitivity). In the same situation, if the target vehicle 902 is not in the passing lane (e.g., if the first lane 904 is to the left of the second lane 906 in a country where vehicles drive on the right side of the road 900), the sensitivity may be lower.
[0171] The sensitivity of the second cut-in sensitivity parameter may further take into account any acceleration or deceleration by the target vehicle 902. For example, if the target vehicle 902 is moving faster than the other vehicles 908 in the first lane 904 and the target vehicle 902 is accelerating, the sensitivity may be high. As another example, if the target vehicle 902 is moving faster than the other vehicles 908 in the first lane 904, and the deceleration of the target vehicle 902 required to avoid a collision with the other vehicle 908 is greater than, for example, 0.1 g, and the target vehicle 902 is not slowing down, the sensitivity may be increased, but not at its highest level. As yet another example, if the target vehicle 902 is moving faster than the other vehicles 908 in the first lane 904, and the deceleration of the target vehicle 902 required to avoid a collision with the other vehicle 908 is greater than, for example, 0.5 g, and the target vehicle 902 is not slowing down, the sensitivity may be at its highest. However, in the same situation, if the closest vehicle preceding vehicle 200 in the second lane 906 is traveling slower than the target vehicle 902 and the deceleration of the target vehicle 902 required to avoid colliding with the closest vehicle preceding vehicle 200 in the second lane 906 is greater than that required to avoid hitting the other vehicle 908, then the sensitivity may be high, but not at its highest. However, in the same situation, if the lane on the other side of the second lane 906 is clear enough to allow for a more gradual deceleration by the target vehicle 902, then the sensitivity may be at its highest because the target vehicle 902 is more likely to cut into the second lane 906 to enter the lane on the other side of the second lane 906. The sensitivity levels described above and throughout the disclosure may exist on a distribution, which may include any number of sensitivity levels, for example, arranged in any desired relative arrangement along the distribution.
[0172] As another example, the sensitivity may be highest if the target vehicle 902 is moving faster than the other vehicles 908 in the first lane 904, the deceleration of the target vehicle 902 required to avoid a collision with the other vehicle 908 is greater than, for example, 0.2 g, and the target vehicle 902 slows down to the speed of the vehicle 200 and / or the closest vehicle preceding the vehicle 200 in the second lane 906. As yet another example, the sensitivity may be lower if the target vehicle 902 is moving faster than the other vehicles 908 in the first lane 904, and the target vehicle 902 slows down to the speed of the vehicle 200 and / or the closest vehicle preceding the vehicle 200 in the second lane 906. As another example, the sensitivity may be lower if the target vehicle 902 is moving faster than the other vehicles 908 in the first lane 904, the deceleration of the target vehicle 902 required to avoid a collision with the other vehicle 908 is less than, for example, 0.2 g, and the target vehicle 902 is slowing down.
[0173] In some embodiments, the vehicle 200 may take into account other actions of the target vehicle 902 when determining the second cut-in sensitivity parameter. For example, if the target vehicle 902 is in the passing lane (e.g., in a country where vehicles drive on the right side of the road 900, and the first lane 904 is to the left of the second lane 906), and another vehicle 908 in the first lane 904 is traveling slower than the target vehicle 902, and the target vehicle 902 is flashing its headlights at the other vehicle 908, a lower sensitivity may be used because the target vehicle 902 has indicated an intention to remain in the first lane 904. In another example, if the target vehicle 902 activates its turn signal, indicating an intention to attempt to cut into the second lane 906, the sensitivity may be higher, but not highest, because the target vehicle 902 is not only indicating an intention to cut into the second lane 906, but is also driving or being driven cautiously. For example, after activation of a turn signal, a cut-in may be detected when the target vehicle 902 exhibits significant lateral movement (eg, 0.5 m towards a lane marker).
[0174] The navigation response taken by vehicle 200 may similarly depend on the presence and actions of vehicles around vehicle 200. For example, in some cases vehicle 200 may slow down to allow target vehicle 902 to cut into second lane 906, while in other situations vehicle 200 may accelerate to allow target vehicle 902 to change into second lane 906 behind vehicle 200. Vehicle 200 may accelerate if, for example, vehicle 200 is traveling below the legal speed limit and vehicle 200 and target vehicle 902 are traveling side by side at approximately the same speed, there is no vehicle preceding vehicle 200 in second lane 906, the nearest vehicle preceding vehicle 200 in second lane 906 is at a safe distance, there is no other vehicle 908 ahead of target vehicle 902 (or the other vehicle 908 is not traveling slower than target vehicle 902), or there is no other open lane into which target vehicle 902 may change. In some cases, vehicle 200 may accelerate more quickly if necessary (eg, if the deceleration required for target vehicle 902 to change into the second lane 906 behind vehicle 200 is greater than, for example, 0.5 g).
[0175] 9C illustrates an example of a predefined cut-in sensitivity modifier in the form of a geographic region. As shown, the target vehicle 902 is traveling in a first lane 904 and the vehicle 200 is traveling in a second lane 906. The geographic region may act as a predefined cut-in sensitivity modifier if the geographic region (e.g., traffic rules and / or driving habits within the geographic region) make the target vehicle 902 more likely to attempt a cut-in than it would be if the target vehicle 902 were not in the geographic region. In some embodiments, the geographic region may include a country or other region having particular legal rules and / or driving habits that govern driving.
[0176] As shown, the geographic region may be detected through detection of road signs 910 (e.g., using monocular and / or stereo image analysis of road signs 910) from which the geographic region may be identified. Alternatively or additionally, in some embodiments, the geographic region may be identified in other ways, such as through one or more geographic indicators or landmarks, using GPS or map data (or other location determination systems associated with vehicle 200), or otherwise. By detecting the geographic region, vehicle 200 may determine whether traffic rules and / or driving habits within the geographic region make it more likely that the target vehicle 902 will attempt to cut in than it would if the target vehicle 902 were not in the geographic region. If so, the geographic region may constitute a predetermined cut-in sensitivity modifier.
[0177] When the predefined cutting-in sensitivity modifier is detected (i.e., when vehicle 200 detects that target vehicle 902 is traveling within the geographic region), vehicle 200 may cause a navigation response based on the identification of the indicator and based on the second cutting-in sensitivity parameter. The navigation response may include, for example, accelerating vehicle 200, decelerating vehicle 200, or (if possible) changing lanes by vehicle 200. The second cutting-in sensitivity parameter may be more sensitive than the first cutting-in sensitivity parameter in the absence of the predefined cutting-in sensitivity modifier because the presence of the cutting-in sensitivity modifier makes it more likely that target vehicle 902 will attempt to cut-in to the second lane 906.
[0178] 9D illustrates an example of a predefined cutting-in sensitivity modifier in the form of a lane closing condition. As shown, the target vehicle 902 is traveling in a first lane 904 and the vehicle 200 is traveling in a second lane 906. The lane closing condition may act as a predefined cutting-in sensitivity modifier when the lane closing condition makes the target vehicle 902 more likely to attempt a cutting-in than the target vehicle 902 would if the first lane 904 had not closed.
[0179] As shown, the lane end condition may be detected through detection of road signs 912 (e.g., using monocular and / or stereoscopic image analysis of the road signs 912) and / or detection of road marks 914 (e.g., using monocular and / or stereoscopic image analysis of the road signs 912) from which the lane end condition may be ascertained. Alternatively or additionally, in some embodiments, the lane end condition may be ascertained in other ways, such as through GPS or map data (or other location system associated with the vehicle 200). By detecting the lane end condition, the vehicle 200 may determine that the target vehicle 902 is more likely to attempt to cut in than it would be if the first lane 904 had not ended. Thus, the lane end condition may constitute a predetermined cut-in sensitivity modifier.
[0180] When the predefined cutting-in sensitivity modifier is detected (i.e., when vehicle 200 detects that first lane 904 is ending), vehicle 200 may cause a navigation response based on the identification of the indicator and based on the second cutting-in sensitivity parameter. The navigation response may include, for example, accelerating vehicle 200, decelerating vehicle 200, or (if possible) changing lanes by vehicle 200. The second cutting-in sensitivity parameter may be more sensitive than the first cutting-in sensitivity parameter in the absence of the predefined cutting-in sensitivity modifier because the presence of the cutting-in sensitivity modifier makes it more likely that the target vehicle 902 will attempt to cut-in to the second lane 906.
[0181] 9E illustrates an example of a predefined cut-in sensitivity modifier in the form of a road split condition. As shown, the target vehicle 902 is traveling in a first lane 904 and the vehicle 200 is traveling in a second lane 906. The road 900 is splitting. The road split condition may act as a predefined cut-in sensitivity modifier if the road split condition makes the target vehicle 902 more likely to attempt to cut-in than it would be if the first lane 904 did not end.
[0182] As shown, the road split condition may be detected through detection of road signs 916 (e.g., using monocular and / or stereo image analysis of road signs 912) and / or detection of road marks 918a, 918b (e.g., using monocular and / or stereo image analysis of road signs 912) from which the road split condition may be ascertained. Alternatively or additionally, in some embodiments, the road split condition may be ascertained in other ways, such as through GPS or map data (or other location system associated with vehicle 200). By detecting the road split condition, vehicle 200 may determine that the target vehicle 902 is more likely to attempt to cut in than the target vehicle 902 would be likely to do if road 900 were not divided. Thus, the road split condition may constitute a predetermined cut-in sensitivity modifier.
[0183] When the predetermined cutting-in sensitivity modifier is detected (i.e., when vehicle 200 detects that road 900 is being divided), vehicle 200 may cause a navigation response based on the identification of the indicator and based on the second cutting-in sensitivity parameter. The navigation response may include, for example, accelerating vehicle 200, decelerating vehicle 200, or (if possible) changing lanes by vehicle 200. The second cutting-in sensitivity parameter may be more sensitive than the first cutting-in sensitivity parameter in the absence of the predetermined cutting-in sensitivity modifier detected because the presence of the cutting-in sensitivity modifier makes it more likely that the target vehicle 902 will attempt to cut into the second lane 906.
[0184] Although specific predetermined cut-in sensitivity modifying factors have been described, it should be understood that other predetermined cut-in sensitivity modifying factors are possible, including any environmental factors that may cause and / or contribute to conditions in which cut-in by the target vehicle 902 is higher or lower than it would otherwise be.
[0185] 10 is a diagram of an exemplary situation in which a vehicle 200 may engage in altruistic behavior, according to disclosed embodiments. As shown, the vehicle 200 may be traveling alongside a target vehicle 1002 on a road 1000. The vehicle 200 may be traveling in a first lane 1004, and the target vehicle 1002 may be traveling in a second lane 1006. Although the first lane 1004 is shown to be the left lane and the second lane 1006 is shown to be the right lane, it should be understood that the first and second lanes 1004 and 1006 may be any adjacent lanes on the road 1000. Furthermore, while only two lanes 1004, 1006 are shown on the road 1000, it should be understood that more lanes are possible. The term "lane" is used for convenience, and in some situations (e.g., when the lanes are not clearly indicated with lane markings), the term "lane" may be understood to refer more generally to the path along which the vehicle 200 and the target vehicle 1002 are traveling.
[0186] Vehicle 200 may be configured to receive images of the environment surrounding road 1000 from one or more image capture devices associated with vehicle 200, such as image capture devices 122, 124, and / or 126. Vehicle 200 may also receive other sensor information, such as GPS or map data. Additionally, in some embodiments, vehicle 200 may receive other sensor information, such as information obtained via a radar or lidar device included in system 100. Based on the images and / or other sensor information, vehicle 200 may detect target vehicle 1002. For example, vehicle 200 may identify an image of target vehicle 1002 in a plurality of images. Vehicle 200 may detect target vehicle 1002 in other ways, including any of the ways described above in connection with cut-in detection module 802.
[0187] Further based on the imagery and / or other sensor information, the vehicle 200 may identify one or more situational features associated with the target vehicle 1002. The situational feature may be any characteristic that indicates, for example, that the target vehicle 1002 would benefit from a lane change into the lane in which the vehicle 200 is traveling. For example, as shown, the situational feature may indicate that the target vehicle 1002 is traveling behind another vehicle 1008 that is traveling slower than the target vehicle 1002. In general, the situational feature may indicate that a cut-in or other navigational response by the target vehicle 1002 may not be necessary, but that a cut-in would be advantageous for the target vehicle 1002.
[0188] In some embodiments, the vehicle 200 may detect the position and / or speed of the target vehicle 1002 as described in connection with FIG. 5B and / or the position of one or more road marks on the road 900 as described in connection with FIG. 5C based on monocular and / or stereo image analysis of the images. As another example, the vehicle 200 may detect that the target vehicle 1002 has initiated a lane change as described above in connection with FIG. 5F. As yet another example, the vehicle 200 may detect that the target vehicle 1002 has initiated a lane change based on analysis of other sensor information, such as information obtained via a radar or lidar device included in the system 100. The vehicle 200 may identify one or more situational features in other ways as well, including any of the ways described above with respect to the altruistic behavior module 806.
[0189] Upon detection of one or more situational features, vehicle 200 may determine a current value associated with an altruistic behavior parameter to determine whether vehicle 200 will allow target vehicle 1002 to cut into first lane 1004. The altruistic behavior parameter may take any of the forms described above in connection with altruistic behavior module 806. For example, the altruistic behavior parameter may be set based on input from an operator of vehicle 200, based on at least one information element identified by analyzing calendar registration information for the operator or passengers of vehicle 200, based on the output of a random number generation function, and / or based on a predetermined number of times a target vehicle is encountered where one or more situational features indicate that the vehicle would benefit from a course change into first lane 1004. The altruistic behavior parameter may be constant or may be updated such that a navigation change is made within vehicle 200 at least a predetermined percentage of encounters with a target vehicle where one or more situational features indicate that the vehicle would benefit from a course change into the path in which vehicle 200 is located. For example, the predetermined percentage can be, for example, at least 10%, at least 20%, at least 30%, etc.
[0190] The vehicle 200 may determine that a change in the navigation state of the vehicle 200 may not be necessary based on one or more situational features associated with the target vehicle 1002. That is, the one or more situational features may indicate that although cutting into the first lane 1004 would be advantageous for the target vehicle 1002, such cutting into may not be necessary (e.g., due to traffic rules or for safety). However, the target vehicle 1002 may, in some examples, make at least one navigational change in the vehicle 200 based on the altruistic behavior parameter and the one or more situational features to allow the target vehicle 1002 to cut into the first lane 1004.
[0191] For example, if the altruistic behavior parameter may be set based on input from an operator of the vehicle 200, the operator may provide an input indicating that the vehicle 200 should allow the target vehicle 1002 to cut in. Based on the altruistic behavior parameter and one or more situational characteristics, the target vehicle 1002 may modify its speed to allow the target vehicle 1002 to cut in into the first lane 1004. As another example, if the altruistic behavior parameter may be set based on the output of a random number generation function, the random number generation function may provide an output indicating that the vehicle 200 should not allow the target vehicle 1002 to cut in into the first lane 1004. For example, the random number generation function may output a binary output (e.g., “no” or “0”), or may output a value output that does not meet a particular threshold (e.g., when the threshold is “>−5”, it may output a “2”). Based on the altruistic behavior parameter and one or more situational features, the target vehicle 1002 may hold its speed to prevent the target vehicle 1002 from cutting into the first lane 1004. As yet another example, if the altruistic behavior parameter is set based on at least one information element identified by analyzing calendar entry information for an operator of the vehicle 200, the operator’s calendar entry information may indicate that the operator will arrive at the destination by a desired time. The altruistic behavior parameter may indicate that the target vehicle 1002 should be allowed to cut in as long as the operator would still reach its destination by its desired time. Based on the altruistic behavior parameter and one or more situational features, the target vehicle 1002 causes navigation changes to be made to allow the target vehicle 1002 to cut in to the first lane 1004 unless doing so would prevent the operator from reaching the destination at the desired time, but not to allow the target vehicle 1002 to cut in to the first lane 1004 if doing so would prevent the operator from reaching the destination at the desired time.
[0192] In some cases, multiple target vehicles may benefit from cutting in to the first lane 1004. For example, as shown, in addition to the target vehicle 1002, an additional target vehicle 1010 may be traveling behind another vehicle 1008 in the second lane 1006. One or more situational characteristics may indicate that cutting in would be advantageous to the additional target vehicle 1010, although cutting in or other navigational response by the additional target vehicle 1010, as with the target vehicle 1002, may not be necessary. For example, as shown, the additional target vehicle 1010 may also be traveling behind the other vehicle 1008, which may be traveling at a slower speed than the additional target vehicle 1010. In these cases, the altruistic behavior parameters may cause the vehicle 200 to treat each of the target vehicle 1002 and the additional target vehicles 1010 the same (i.e., allow both or neither to cut into the first lane 1004), or the altruistic behavior parameters may cause the vehicle 200 to treat the target vehicle 1002 and the additional target vehicles 1010 differently.
[0193] For example, the altruistic behavior parameter may be set based on the number of encounters with target vehicles, such as vehicles 1002, 1010, where situational characteristics indicate that such vehicles would benefit from a course change into the first lane 1004. For example, in the case of two target vehicles 1002, 1010, the altruistic behavior parameter may cause the vehicle 200 to only allow the target vehicle 1002 to cut into the first lane 1004, but not the additional target vehicles 1010 (whereas the altruistic behavior parameter may have allowed both the target vehicle 1002 and the additional target vehicles 1010 to cut in if encountered alone).
[0194] As another example, in these cases, the altruistic behavior parameter may be updated to cause a navigational change in the vehicle 200 to occur in at least a predetermined percentage of encounters with a target vehicle 1002, 1010 where one or more situational characteristics indicate that the vehicle 200 would benefit from a course change into the first lane 1004. For example, the altruistic behavior parameter may specify that the vehicle 200 should perform a navigational change in at least 10% of the encounters. Based on the altruistic behavior parameter, the vehicle 200 may accept one or both of the target vehicle 1002 and the additional target vehicles 1010, depending on the overall percentage of encounters that are determined on a continuous basis, as long as the vehicle 200 performs a navigational change in at least 10% of the encounters.
[0195] Additionally, although consideration of altruistic behavior parameters was discussed above in connection with the example shown in FIG. 10, altruistic behavior parameters may be considered in any of the examples discussed above in connection with FIGS. 9A-9E.
[0196] 11 is a flow chart illustrating an example process 1100 for vehicle interruption detection and response, according to disclosed embodiments. In some embodiments, processing unit 110 of system 100 may execute one or more of modules 402-408 and 802-806. In other embodiments, instructions stored in one or more of modules 402-408 and 802-806 may be executed remotely from system 100 (e.g., via a server accessible over a network via wireless transceiver 172). In yet other embodiments, instructions associated with one or more of modules 402-408 and 802-806 may be executed by processing unit 110 and a remote server.
[0197] As shown, process 1100 includes receiving an image at step 1102. For example, vehicle 200 may receive a plurality of images from at least one image capture device (e.g., image capture devices 122, 124, 126) associated with vehicle 200 via a data interface. As mentioned above, in other embodiments, as an alternative or in addition to analyzing the images, vehicle 200 may instead analyze other sensor information, such as information obtained via a radar or lidar device included in system 100.
[0198] At step 1104, process 1100 includes identifying a target vehicle. For example, vehicle 200 may identify, in the plurality of images, an image of a target vehicle traveling in a first lane that is different from a second lane in which vehicle 200 is traveling. Identifying the image of the target vehicle may include monocular or stereo image analysis and / or other sensor information, for example, as described above in connection with detection module 802. In other embodiments, as previously described, identifying the image of the target vehicle may include analyzing other sensor information, such as information obtained via a radar or lidar device included in system 100, as an alternative or in addition to analyzing the images.
[0199] At step 1106, process 1100 includes identifying an indicator that the target vehicle will change lanes. For example, vehicle 200 may identify at least one indicator that the target vehicle will change from a first lane to a second lane based on analysis of the multiple images. Identifying the indicator may include, for example, monocular or stereo image analysis and / or other sensor information (e.g., radar or lidar data), as described above in connection with cut-in detection module 802.
[0200] At step 1108, process 1100 includes determining whether a predefined cut-in sensitivity modifier is present. For example, vehicle 200 may determine whether at least one predefined cut-in sensitivity modifier is present in the environment of vehicle 200. The at least one predefined cut-in sensitivity modifier may take any of the forms described above in connection with, for example, FIGS. 9A-9E. The predefined cut-in sensitivity modifier may include, for example, a lane end condition, an obstacle in the path of the target vehicle, a road division, or a geographic region. Detecting the at least one predefined cut-in sensitivity modifier may include, for example, monocular or stereo image analysis and / or other sensor information (e.g., radar or lidar data), as described above in connection with cut-in detection module 802.
[0201] If the predefined intrusion sensitivity modifier is not detected, process 1100 may proceed to step 1110 and generate a first navigational response based on the indicator and the first intrusion sensitivity parameter. For example, vehicle 200 may generate a first navigational response in the vehicle based on the identification of the at least one indicator and based on the first intrusion sensitivity parameter if the intrusion sensitivity modifier is not detected. The first intrusion sensitivity parameter may take any of the forms described above in connection with intrusion response module 804. Further, as previously mentioned, in some embodiments, step 1110 may instead include generating a first navigational response based on the indicator and a value associated with the first predefined intrusion sensitivity parameter, where the first navigational response may take any of the forms described for the navigational response with respect to FIG. 4.
[0202] On the other hand, if at least one predefined intrusion sensitivity modifier is detected, process 1100 may proceed to step 1112 and generate a second navigation response based on the indicator and the second intrusion sensitivity parameter. For example, vehicle 200 may generate a second navigation response in the vehicle based on the identification of the at least one indicator if an intrusion sensitivity modifier is detected and based on the second intrusion sensitivity parameter. The second intrusion sensitivity parameter may take any of the forms described above in connection with intrusion response module 804. The second intrusion sensitivity parameter may be different (e.g., more sensitive) than the first intrusion sensitivity parameter. Further, as previously mentioned, in some embodiments, step 1112 may instead include generating a second navigation response based on the indicator and a value associated with the second predefined intrusion sensitivity parameter. The second navigation response may take any of the forms described for the navigation response with respect to FIG. 4.
[0203] 12 is a flow chart illustrating an example process 1200 for navigating while taking into account altruistic behavior, according to disclosed embodiments. In some embodiments, processing unit 110 of system 100 may execute one or more of modules 402-408 and 802-806. In other embodiments, instructions stored in one or more of modules 402-408 and 802-806 may be executed remotely from system 100 (e.g., via a server accessible over a network via wireless transceiver 172). In yet other embodiments, instructions associated with one or more of modules 402-408 and 802-806 may be executed by processing unit 110 and a remote server.
[0204] As shown, process 1200 includes receiving an image at step 1202. For example, vehicle 200 may receive a plurality of images from at least one image capture device (e.g., image capture devices 122, 124, 126) associated with vehicle 200 via a data interface. As mentioned above, in other embodiments, as an alternative or in addition to analyzing the images, vehicle 200 may instead analyze other sensor information, such as information obtained via a radar or lidar device included in system 100.
[0205] Process 1200 includes identifying a target vehicle at step 1204. For example, vehicle 200 may identify at least one target vehicle in the environment of vehicle 200 in a plurality of images. Identifying the target vehicle may include monocular or stereo image analysis and / or other sensor information, for example, as described above in connection with altruistic behavior module 806.
[0206] At step 1206, process 1200 includes identifying one or more situational features associated with the target vehicle. For example, vehicle 200 may identify one or more situational features associated with the target vehicle based on an analysis of the multiple images. The situational features may take any of the forms described above in connection with altruistic behavior module 806 and / or FIGS. 9A-9E and 10. For example, the situational features may indicate that the target vehicle would benefit from a course change to the path of vehicle 200. As another example, the situational features may indicate that the target vehicle is traveling in a lane adjacent to the lane in which vehicle 200 is traveling, and that the target vehicle is behind a vehicle traveling slower than the target vehicle and slower than vehicle 200. Identifying the situational features may include monocular or stereo image analysis and / or other sensor information (e.g., radar or lidar data), for example, as described above in connection with altruistic behavior module 806.
[0207] At step 1208, process 1200 includes determining a current value of the altruistic behavior parameter. For example, vehicle 200 may determine a current value associated with the altruistic behavior parameter. The altruistic behavior parameter may take any of the forms described above in connection with altruistic behavior module 806 and / or FIG. 10, and determining the altruistic behavior parameter may be performed in any of the ways described above in connection with altruistic behavior module 806 and / or FIG. 10. The value of the altruistic behavior parameter may be set based on input from an operator of vehicle 200, based on at least one information element determined by analyzing calendar entry information for the operator of vehicle 200, based on the output of a random number generation function, and / or based on a predetermined number of encounters with such target vehicles where one or more situational characteristics indicate that the vehicle 200 would benefit from a course change to its path. Alternatively or additionally, the value of the altruistic behavior parameter may be updated so that a navigation change occurs within vehicle 200 in at least a predetermined percentage (e.g., 10%, 20%, 30%, etc.) of encounters with a target vehicle where one or more situational characteristics indicate that the vehicle would benefit from a course change to the path on which vehicle 200 is located.
[0208] At step 1210, process 1200 includes effecting a navigation change based on the one or more situational features and the current value of the altruistic behavior parameter. For example, vehicle 200 may determine, based on one or more situational features associated with the target vehicle, that a change in the navigational state of vehicle 200 may not be necessary, but may still effect at least one navigation change in vehicle 200 based on the current value associated with the altruistic behavior parameter and based on the one or more situational features associated with the target vehicle. Effecting the navigation change may be performed by altruistic behavior module 806 and / or any of the methods described above in connection with FIG. 10.
[0209] As mentioned above, in any process or process step described in this disclosure, in addition to performing image analysis of images captured from one or more front and / or rear cameras, system 100 (included in vehicle 200) may also analyze other sensor information, such as information obtained via radar and / or lidar devices.
[0210] The above description is provided for illustrative purposes. It is not exhaustive and is not limited to the precise form or embodiment disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of this specification and practice of the disclosed embodiments. Furthermore, although aspects of the disclosed embodiments are described as being stored in memory, those skilled in the art will appreciate that these aspects can also be stored on other types of computer readable media, such as secondary storage devices, for example, hard disks or CD ROMs or other forms of RAM or ROM, USB media, DVDs, Blu-rays, 4K Ultra HD Blu-rays, or other optically driven media.
[0211] Computer programs based on the written description and the disclosed methods are within the skill of an experienced developer. The various programs or program modules can be created using any technique known to those skilled in the art or can be designed in conjunction with existing software. For example, program sections or program modules can be designed in or with the .Net Framework, .Net Compact Framework (and related languages such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML / AJAX combination, XML, or HTML including Java applets.
[0212] Furthermore, while exemplary embodiments have been described herein, the scope of any embodiment includes equivalent elements, modifications, omissions, combinations (e.g., aspects across various embodiments), adaptations, and / or substitutions as would be understood by one of ordinary skill in the art based on this disclosure. Limitations in the claims should be interpreted broadly based on the language used in the claims, and not limited to the examples described herein or examples in the practice of this application. Examples should be construed as non-exclusive. Furthermore, the steps of the disclosed methods may be modified in any manner, including reordering steps and / or inserting or deleting steps. Thus, it is intended that the specification and examples be considered as merely illustrative, with the true scope and spirit being indicated by the following claims and their full scope of equivalents.
Claims
1. 1. A vehicle navigation system, comprising: a memory having instructions; A circuit, which, during operation, in response to the instructions: identifying a target vehicle in an environment of a vehicle including the vehicle navigation system, the target vehicle traveling in a lane adjacent to a lane in which the vehicle is traveling; receiving image data of the target vehicle from at least one image capture device of the vehicle; identifying one or more situational characteristics of the target vehicle based on analysis of the image data, the situational characteristics of the target vehicle including an indication that the target vehicle has an activated turn signal, an indication that an additional target vehicle is traveling ahead of the target vehicle, and an indication that the additional target vehicle is traveling at a slower speed than the target vehicle; changing a navigation state of the vehicle to permit the target vehicle to act, the vehicle being configured to effect the change in the navigation state based on a determination that the one or more situational characteristics, including the active turn indicator, indicate that the target vehicle would benefit from the change in the navigation state; The circuitry is configured to perform A vehicle navigation system comprising:
2. The vehicle navigation system of claim 1 , wherein the one or more situational characteristics indicate that the target vehicle would benefit from a course change to the vehicle's route.
3. The vehicle navigation system of claim 2 , wherein the one or more situational features indicate that the target vehicle has moved laterally toward the path of the vehicle.
4. The vehicle navigation system of claim 1 , wherein the one or more situational features indicate that the target vehicle is traveling in a lane adjacent to a lane in which the vehicle is traveling.
5. 5. The vehicle navigation system of claim 1, wherein the action is cutting into a lane in which the vehicle is traveling.
6. The vehicle navigation system of claim 5 , wherein the action is a cut-in into the lane in which the vehicle is traveling and occurs at an interchange.
7. 7. A vehicle navigation system according to claim 1, wherein the circuitry is configured to change a speed of the vehicle in order to change the navigation state of the vehicle.
8. 8. A vehicle navigation system as claimed in any preceding claim, wherein the circuitry is configured to use a radar of the vehicle to identify the target vehicle within the environment of the vehicle.
9. A vehicle, at least one image capture device; a vehicle navigation system, identifying a target vehicle in an environment of the vehicle traveling in a lane adjacent to a lane in which the vehicle is traveling; receiving image data of the target vehicle from the at least one image capture device; identifying one or more situational characteristics of the target vehicle based on analysis of the image data, the situational characteristics of the target vehicle including an indication that the target vehicle has an activated turn signal, an indication that an additional target vehicle is traveling ahead of the target vehicle, and an indication that the additional target vehicle is traveling at a slower speed than the target vehicle; changing a navigation state of the vehicle to enable a behavior of the target vehicle, the vehicle being configured to effect the change in the navigation state based on a determination that the one or more situational characteristics, including the active turn indicator, indicate that the target vehicle would benefit from the change in the navigation state; configured to execute vehicle.
10. The vehicle of claim 9 , wherein the one or more situational characteristics indicate that the target vehicle would benefit from a course change to the vehicle's path.
11. The vehicle of claim 10 , wherein the one or more situational features indicate that the target vehicle has moved laterally toward the path of the vehicle.
12. 12. The vehicle of claim 9, wherein the one or more situational features indicate that the target vehicle is traveling in a lane adjacent to a lane in which the vehicle is traveling.
13. 13. The vehicle of claim 9, wherein the action is cutting into a lane in which the vehicle is traveling.
14. The vehicle of claim 13 , wherein the action is cutting into the lane in which the vehicle is traveling and occurs at an interchange.
15. 15. A vehicle as claimed in any one of claims 9 to 14, wherein the vehicle navigation system is configured to vary a speed of the vehicle in order to change the navigation state of the vehicle.
16. 16. A vehicle as claimed in any one of claims 9 to 15, further comprising a radar, and wherein the vehicle navigation system is configured to use the vehicle's radar to identify the target vehicle in the environment of the vehicle.
17. When executed by a vehicle's computer, the computer: identifying a target vehicle in the vehicle's environment traveling in a lane adjacent to the lane in which the vehicle is traveling; receiving image data of the target vehicle from at least one image capture device of the vehicle; identifying one or more situational characteristics of the target vehicle based on analysis of the image data, the situational characteristics of the target vehicle including an indication that the target vehicle has an activated turn signal, an indication that an additional target vehicle is traveling ahead of the target vehicle, and an indication that the additional target vehicle is traveling at a slower speed than the target vehicle; and a procedure for changing a navigation state of the vehicle to enable the target vehicle's action, the vehicle being configured to effect the change in the navigation state based on a determination that the one or more situational characteristics, including the active turn indicator, indicate that the target vehicle would benefit from the change in the navigation state.
18. 20. The program of claim 17, wherein the one or more situational characteristics indicate that the target vehicle would benefit from a course change to the vehicle's path.
19. The method of claim 18 , wherein the one or more situational features indicate that the target vehicle has moved laterally toward the path of the vehicle.
20. 20. The program of claim 17, wherein the one or more situational features indicate that the target vehicle is traveling in a lane adjacent to a lane in which the vehicle is traveling.
21. 21. The program of claim 17, wherein the action is cutting into a lane in which the vehicle is traveling.
22. 22. The program of claim 21, wherein the action is a cut-in into the lane in which the vehicle is traveling and occurs at an interchange.
23. 23. The method of claim 17, wherein changing the navigation state of the vehicle comprises changing a speed of the vehicle.
24. 24. The program of claim 17, wherein identifying the target vehicle in the environment of the vehicle includes using a radar of the vehicle.
Citation Information
Patent Citations
Inter-vehicle communication device and inter-vehicle communication system
JP2008158588A
Inter-vehicle distance control unit
JP2014148293A
Vehicle platoon controller, and vehicle platoon control method
JP2015022421A
