Systems, methods and computer programs for navigating a vehicle

Abstract

A system and method for navigating an autonomous vehicle is provided. [Solution] The system includes at least one processing device programmed to receive at least one image related to an environment of a host vehicle from an image capture device, analyze the at least one image to identify a side of a parked vehicle, identify a door edge of the parked vehicle within the at least one image, determine changes in image characteristics of the door edge of the parked vehicle based on analysis of one or more subsequent images received from the image capture device, and alter a navigation route of the host vehicle based at least in part on the changes in the image characteristics of the door edge of the parked vehicle.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 354,946, filed June 27, 2016, and U.S. Provisional Patent Application No. 62 / 445,500, filed January 12, 2017. The above applications are incorporated herein by reference in their entireties.

[0002] background Technical Field The present disclosure relates generally to autonomous vehicle navigation. In addition, the present disclosure relates to a system and method for navigating a host vehicle based on detecting a door opening, a system and method for navigating a host vehicle based on detecting a target vehicle entering the host vehicle's lane, a system and method for navigating a host vehicle based on detecting whether the road on which the host vehicle is traveling is a one-way road, and a system and method for determining a predicted state of a parked vehicle. [Background technology]

[0003] Background information As technology continues to evolve, the goal of fully autonomous vehicles capable of navigating roads becomes more realistic. An autonomous vehicle may need to consider various factors and, based on those factors, make appropriate decisions to safely and accurately reach its intended destination. For example, an autonomous vehicle may need to process and interpret information from visual information (e.g., information captured from a camera), radar, or lidar, and may also use information obtained from other sources (e.g., from a GPS device, speed sensors, accelerometers, suspension sensors, etc.). At the same time, to navigate to its destination, an autonomous vehicle may also need to identify its location within a particular road (e.g., a particular lane within a multi-lane road), navigate alongside other vehicles, avoid obstacles and pedestrians, observe traffic signals and signs, steer from one road to another at appropriate intersections or interchanges, and respond to any other conditions that arise or occur during the vehicle's operation.

[0004] An autonomous vehicle must be able to react to changing conditions with sufficient time to adjust the vehicle's navigation path or apply the brakes. Many conventional algorithms, such as those used in existing autonomous braking systems, do not have reaction times comparable to humans. Therefore, they are often better suited to be used as a backup for a human driver than for use in a fully autonomous vehicle.

[0005] Furthermore, characteristics of parked vehicles are often good indicators of road characteristics. For example, the direction of parked vehicles can indicate whether the road is a one-way street, and the spacing between vehicles can indicate whether pedestrians are likely to emerge between the vehicles. However, existing autonomous vehicle algorithms do not use such characteristics.

[0006] Finally, autonomous vehicle systems can use measurements that are not available to a human driver. For example, an autonomous vehicle system can use an infrared camera to assess the environment and make predictions. However, many conventional systems do not utilize a combination of measurements, such as a visual camera and an infrared camera. Embodiments of the present disclosure can address one or more of the above-mentioned shortcomings of conventional systems. Summary of the Invention

[0007] overview Embodiments according to the present disclosure provide systems and methods for autonomous vehicle navigation. The disclosed embodiments may provide autonomous vehicle navigation features using cameras. For example, according to the disclosed embodiments, the disclosed system may include one, two, or three or more cameras that monitor the vehicle's environment. The disclosed system may provide a navigation response, for example, based on an analysis of images captured by one or more of the cameras. Some embodiments may further include one, two, or three or more infrared cameras that monitor the environment. Thus, some embodiments may provide a navigation response, for example, based on an analysis of a visual image, an infrared image, or any combination thereof.

[0008] The navigation response may also take into account other data, including, for example, global positioning system (GPS) data, sensor data (eg, from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data.

[0009] In one embodiment, a system for navigating a vehicle based on detecting a door-opening event in an environment of a host vehicle may include at least one processing device. The at least one processing device may be programmed to receive at least one image related to the environment of the host vehicle from an image capture device and analyze the at least one image to identify a side of the parked vehicle. The at least one processing device may be further programmed to identify, within the at least one image, a first structural feature of the parked vehicle in a region forward of the side of the parked vehicle and a second structural feature of the parked vehicle in a region rearward of the side of the parked vehicle, and identify, within the at least one image, a door edge of the parked vehicle near the first structural feature and the second structural feature. The at least one processing device may also be programmed to determine a change in image characteristics of the door edge of the parked vehicle based on an analysis of one or more subsequent images received from the image capture device and alter a navigation route of the host vehicle based at least in part on the change in image characteristics of the door edge of the parked vehicle.

[0010] In another embodiment, a method for navigating a vehicle based on detecting a door-opening event in an environment of a host vehicle may include receiving at least one image related to an environment of the host vehicle from an image capture device and analyzing the at least one image to identify a side of the parked vehicle. The method may further include identifying, in the at least one image, a first structural feature of the parked vehicle in a region forward of the side of the parked vehicle and a second structural feature of the parked vehicle in a region rearward of the side of the parked vehicle, and identifying, in the at least one image, a door edge of the parked vehicle near the first structural feature and the second structural feature. The method may also include determining a change in an image characteristic of the door edge of the parked vehicle based on an analysis of one or more subsequent images received from the image capture device, and altering a navigation route of the host vehicle based at least in part on the change in the image characteristic of the door edge of the parked vehicle.

[0011] In yet another embodiment, a system for navigating a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is traveling may include at least one processing device. The at least one processing device may be programmed to receive a plurality of images related to an environment of the host vehicle from an image capture device and to analyze at least one of the plurality of images to identify the target vehicle and at least one wheel component on a side of the target vehicle. The at least one processing device may be further programmed to analyze an area including the at least one wheel component of the target vehicle in at least two of the plurality of images to identify movement associated with the at least one wheel component of the target vehicle, and to effect at least one navigational change of the host vehicle based on the identified movement associated with the at least one wheel component of the target vehicle.

[0012] In yet another embodiment, a method for navigating a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is traveling may include receiving a plurality of images related to an environment of the host vehicle from an image capture device and analyzing at least one of the plurality of images to identify the target vehicle and at least one wheel component on a side of the target vehicle. The method may further include analyzing an area including the at least one wheel component of the target vehicle in at least two of the plurality of images to identify movement associated with the at least one wheel component of the target vehicle, and causing at least one navigational change of the host vehicle based on the identified movement associated with the at least one wheel component of the target vehicle.

[0013] In yet another embodiment, a system for detecting whether a road on which a host vehicle travels is a one-way road may include at least one processing device. The at least one processing device may be programmed to receive at least one image related to an environment of the host vehicle from an image capture device, identify a first plurality of vehicles on a first side of the road on which the host vehicle travels based on an analysis of the at least one image, and identify a second plurality of vehicles on a second side of the road on which the host vehicle travels based on an analysis of the at least one image. The at least one processing device may be further programmed to determine a first facing direction associated with the first plurality of vehicles, determine a second facing direction associated with the second plurality of vehicles, and cause at least one navigation change of the host vehicle if both the first facing direction and the second facing direction are opposite to a direction of travel of the host vehicle.

[0014] In yet another embodiment, a method for detecting whether a road on which a host vehicle is traveling is a one-way road may include receiving at least one image from an image capture device related to an environment of the host vehicle, identifying a first plurality of vehicles on a first side of the road on which the host vehicle is traveling based on an analysis of the at least one image, and identifying a second plurality of vehicles on a second side of the road on which the host vehicle is traveling based on an analysis of the at least one image. The method may further include determining a first forward direction associated with the first plurality of vehicles, determining a second forward direction associated with the second plurality of vehicles, and causing at least one navigation change of the host vehicle if both the first forward direction and the second forward direction are opposite to a direction of travel of the host vehicle.

[0015] In another embodiment, a system for navigating a host vehicle may include at least one processing device. The at least one processing device may be programmed to receive navigation instructions for navigating the host vehicle from a first road along which the host vehicle is traveling to a second road and to receive at least one image from an image capture device related to an environment of the second road. The at least one processing device may be further programmed to identify a first plurality of vehicles on a first side of the second road based on analysis of the at least one image and to identify a second plurality of vehicles on a second side of the second road based on analysis of the at least one image. The at least one processing device may also be programmed to determine a first forward direction associated with the first plurality of vehicles, determine a second forward direction associated with the second plurality of vehicles, and determine that both the first forward direction and the second forward direction are opposite to a direction of travel along which the host vehicle will travel if the host vehicle enters the second road. The at least one processing device may be further programmed to interrupt the navigation instructions in response to determining that both the first forward direction and the second forward direction are opposite to a direction of travel in which the host vehicle would travel if the host vehicle were navigated onto the second road.

[0016] In yet another embodiment, a system for determining a predicted state of a parked vehicle within an environment of a host vehicle may include an image capture device, an infrared image capture device, and at least one processing device. The at least one processing device may be programmed to receive a plurality of images related to the environment of the host vehicle from the image capture device, analyze at least one of the plurality of images to identify the parked vehicle, and analyze at least two of the plurality of images to identify a change in lighting conditions of at least one light associated with the parked vehicle. The at least one processing device may be further programmed to receive at least one thermal image of the parked vehicle from the infrared image capture device, determine a predicted state of the parked vehicle based on an analysis of the change in lighting conditions and the at least one thermal image, and cause at least one navigation response by the host vehicle based on the predicted state of the parked vehicle.

[0017] In yet another embodiment, a method for determining a predicted state of a parked vehicle within an environment of a host vehicle may include receiving a plurality of images related to an environment of the host vehicle from an image capture device, analyzing at least one of the plurality of images to identify the parked vehicle, and analyzing at least two of the plurality of images to identify a change in lighting conditions of at least one light associated with the parked vehicle. The method may further include receiving at least one thermal image of the parked vehicle from an infrared image capture device, determining a predicted state of the parked vehicle based on the change in lighting conditions and an analysis of the at least one thermal image, and causing at least one navigation response by the host vehicle based on the predicted state of the parked vehicle.

[0018] In yet another embodiment, a system for determining a predicted state of a parked vehicle within an environment of a host vehicle may include an image capture device and at least one processing device. The at least one processing device may be programmed to receive a plurality of images related to the environment of the host vehicle from the image capture device. The at least one processing device may be further programmed to analyze at least one of the plurality of images to identify the parked vehicle and to analyze at least two of the plurality of images to identify a change in lighting conditions of at least one light associated with the parked vehicle. The at least one processing device may also be programmed to determine a predicted state of the parked vehicle based on the change in lighting conditions and to cause at least one navigation response by the host vehicle based on the predicted state of the parked vehicle.

[0019] In yet another embodiment, a system for navigating a host vehicle may include at least one processing device. The at least one processing device may be programmed to receive a plurality of images from a camera representing an environment of the host vehicle and to analyze at least one of the plurality of images to identify at least two stopped vehicles. The at least one processing device may be further programmed to determine a separation between the two stopped vehicles based on the analysis of at least one of the plurality of images, and to effect at least one navigation change of the host vehicle based on the determined separation between the two stopped vehicles.

[0020] In yet another embodiment, a method for navigating a host vehicle may include receiving a plurality of images from a camera representing an environment of the host vehicle and analyzing at least one of the plurality of images to identify at least two stopped vehicles. The method may further include determining a spacing between the two stopped vehicles based on the analysis of at least one of the plurality of images, and making at least one navigation change of the host vehicle based on the determined spacing between the two stopped vehicles.

[0021] According to other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that are executed by at least one processing device and that perform any of the methods described herein.

[0022] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the scope of the claims.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a diagrammatic representation of an exemplary system according to the disclosed embodiments. [Figure 2A] 1 is a side view representation of an exemplary vehicle including a system according to disclosed embodiments. [Figure 2B] 2B is a top view representation of the vehicle and system shown in FIG. 2A according to a disclosed embodiment. [Figure 2C] 1 is a top view representation of another embodiment of a vehicle including a system according to the disclosed embodiments. [Figure 2D] 1 is a top view representation of yet another embodiment of a vehicle including a system according to the disclosed embodiments. [Figure 2E] 1 is a top view representation of yet another embodiment of a vehicle including a system according to the disclosed embodiments. [Figure 2F] 1 is a diagrammatic representation of an exemplary vehicle control system according to the disclosed embodiments. [Figure 3A] 1 is a diagrammatic representation of the interior of a vehicle including a rearview mirror and a user interface of a vehicle imaging system according to disclosed embodiments. [Figure 3B] 1 is a diagram of an example of a camera mount configured to be positioned behind a rearview mirror, facing a vehicle windshield, according to a disclosed embodiment. [Figure 3C]3C is a diagram of the camera mount shown in FIG. 3B from a different perspective, according to a disclosed embodiment. [Figure 3D] 1 is a diagram of an example of a camera mount configured to be positioned behind a rearview mirror, facing a vehicle windshield, according to a disclosed embodiment. [Figure 4] FIG. 1 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments. [Figure 5A] 1 is a flowchart illustrating an exemplary process for generating one or more navigational responses based on monocular image analysis, according to disclosed embodiments. [Figure 5B] 1 is a flowchart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a set of images, according to disclosed embodiments. [Figure 5C] 1 is a flowchart illustrating an exemplary process for detecting road markings and / or lane geometry information in a set of images, according to disclosed embodiments. [Figure 5D] 1 is a flowchart illustrating an exemplary process for detecting traffic lights in a set of images, according to disclosed embodiments. [Figure 5E] 1 is a flowchart of an exemplary process for generating one or more navigational responses based on a vehicle path, according to disclosed embodiments. [Figure 5F] 1 is a flowchart illustrating an example process for determining whether a leading vehicle is changing lanes, according to disclosed embodiments. [Figure 6] 1 is a flowchart illustrating an example process for generating one or more navigational responses based on stereo image analysis, according to disclosed embodiments. [Figure 7] 1 is a flowchart illustrating an exemplary process for generating one or more navigational responses based on an analysis of three sets of images, according to disclosed embodiments. [Figure 8]FIG. 10 is another example block diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments. [Figure 9] FIG. 1 is a schematic diagram of a road from the perspective of a system in accordance with a disclosed embodiment; [Figure 10] FIG. 10 is another schematic diagram of a road from the perspective of a system according to the disclosed embodiments. [Figure 11] FIG. 1 is a schematic diagram of a parked car from the perspective of a system according to a disclosed embodiment. [Figure 12A] FIG. 1 is a schematic diagram of a door opening event from the perspective of the system according to the disclosed embodiments. [Figure 12B] FIG. 10 is another schematic diagram of a door opening event from the perspective of the system according to the disclosed embodiments. [Figure 13] FIG. 10 is another schematic diagram of a road from the perspective of a system according to the disclosed embodiments. [Figure 14] 1 is a flowchart illustrating an example process for generating one or more navigational responses based on detecting a door opening event, according to disclosed embodiments. [Figure 15] FIG. 10 is another example block diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments. [Figure 16A] FIG. 1 is a schematic diagram of a parked car from the perspective of a system according to a disclosed embodiment. [Figure 16B] FIG. 10 is another schematic diagram of a parked car from the perspective of the system according to the disclosed embodiments. [Figure 17] 1 is a flowchart illustrating an example process for generating one or more navigational responses based on detecting a target vehicle entering a lane of a host vehicle, according to disclosed embodiments. [Figure 18] 1 is a flowchart illustrating an exemplary process for warping a road homography, in accordance with the disclosed embodiments. [Figure 19]FIG. 10 is another example block diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments. [Figure 20A] FIG. 1 is a schematic diagram of a one-way road from the perspective of a system in accordance with a disclosed embodiment. [Figure 20B] FIG. 10 is another schematic diagram of a one-way road from the perspective of a system according to the disclosed embodiments. [Figure 21] 1 is a flowchart illustrating an example process for generating one or more navigational responses based on detecting whether a road on which a host vehicle is traveling is a one-way road, according to disclosed embodiments. [Figure 22] FIG. 10 is another example block diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments. [Figure 23A] FIG. 1 is a schematic diagram of a parked car from the perspective of a system according to a disclosed embodiment. [Figure 23B] FIG. 1 is a schematic diagram of a parked car with varying lighting from the perspective of a system in accordance with the disclosed embodiments. [Figure 23C] FIG. 1 is a schematic diagram of a heat map of a parked car from the perspective of a system in accordance with a disclosed embodiment; [Figure 24] 1 is a flowchart illustrating an exemplary process for determining a predicted state of a parked vehicle, according to disclosed embodiments. [Figure 25] 1 is a flowchart illustrating an exemplary process for aligning visible light and infrared images from a system according to disclosed embodiments. [Figure 26] FIG. 10 is another example block diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments. [Figure 27A] FIG. 10 is another schematic diagram of a road from the perspective of a system according to the disclosed embodiments. [Figure 27B] FIG. 10 is another schematic diagram of a road with detection hotspots according to the disclosed embodiments; [Figure 28]1 is a flowchart illustrating an exemplary process for navigating a host vehicle, according to disclosed embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0025] Detailed Description The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components shown in the drawings, and the exemplary methods described herein may be modified by substituting, reordering, deleting, or adding steps of the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the appropriate scope is defined by the appended claims.

[0026] Autonomous Vehicle Overview As used throughout this disclosure, the term “autonomous vehicle” refers to a vehicle capable of making at least one navigation change without driver input. A “navigation change” refers to one or more changes in the vehicle's steering, braking, or acceleration / deceleration. To be autonomous, a vehicle need not be fully automatic (e.g., fully operable without a driver or driver input). Rather, an autonomous vehicle includes a vehicle that can operate under driver control during certain periods of time and without driver control during other periods of time. An autonomous vehicle can also include a vehicle that controls only some aspects of vehicle navigation, such as steering (e.g., to maintain the vehicle course between vehicle lane constraints) or some steering maneuvers under certain circumstances (though not under all circumstances), but may leave other aspects (e.g., braking or braking under certain circumstances) to the driver. In some cases, an autonomous vehicle may handle some or all aspects of the vehicle's braking, speed control, and / or steering.

[0027] Because human drivers generally rely on visual cues and observations to control their vehicles, transportation infrastructure is built accordingly, with lane markings, traffic signs, and traffic lights designed to provide visual information to drivers. In light of these design features of transportation infrastructure, autonomous vehicles may include a camera and a processing unit that analyzes visual information captured from the vehicle's environment. The visual information may include, for example, images depicting transportation infrastructure components (e.g., lane markings, traffic signs, traffic lights, etc.) and other obstacles (e.g., other vehicles, pedestrians, debris, etc.) observable by the driver. The autonomous vehicle may also include an infrared camera. In such an embodiment, the processing unit may analyze thermal information captured from the environment, either individually or in combination with the visual information.

[0028] Additionally, autonomous vehicles may use stored information, such as information that provides a model of the vehicle's environment as it navigates. For example, a vehicle may use GPS data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data to provide information related to the vehicle's environment while the vehicle is traveling, and the vehicle (and other vehicles) may use that information to locate itself on the model. Some vehicles may also be capable of vehicle-to-vehicle communication, sharing information, alerting peer vehicles of hazards or changes in the vehicle's surroundings, etc.

[0029] System Overview FIG. 1 is a block diagram representation of system 100 according to an exemplary embodiment. System 100 may include various components depending on the requirements of a particular implementation. In some embodiments, system 100 may include a processing unit 110, an image acquisition unit 120, a position sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. Processing unit 110 may include one or more processing devices. In some embodiments, processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, image acquisition unit 120 may include any number of image acquisition devices and components depending on the requirements of a particular application. In some embodiments, image acquisition unit 120 may include one or more image capture devices (e.g., cameras, CCDs, or any other type of image sensor), such as image capture device 122, image capture device 124, image capture device 126, etc. In some embodiments, image acquisition unit 120 may further include one or more infrared capture devices (e.g., an infrared camera, a far-infrared (FIR) detector, or any other type of infrared sensor), for example, one or more of image capture device 122, image capture device 124, and image capture device 126 may include an infrared image capture device.

[0030] System 100 may also include a data interface 128 that communicatively connects processing unit 110 to image acquisition unit 120. For example, data interface 128 may include any one or more wired and / or wireless links for transmitting image data acquired by image acquisition unit 120 to processing unit 110.

[0031] Wireless transceiver 172 may include one or more devices configured to exchange transmissions with one or more networks (e.g., cellular, Internet, etc.) over a wireless interface through the use of radio frequencies, infrared frequencies, magnetic fields, or electric fields. Wireless transceiver 172 may send and / or receive data using any known standard (e.g., Wi-Fi, Bluetooth, Bluetooth Smart, 802.15.4, ZigBee, etc.). Such transmissions may include communications from the host vehicle to one or more remotely located servers. Such transmissions may include communications (one-way or two-way) between the host vehicle and one or more target vehicles in the host vehicle's environment (e.g., to facilitate adjusting the host vehicle's navigation in light of or with the target vehicles in the host vehicle's environment), as well as broadcast transmissions to unspecified recipients in the vicinity of the transmitting vehicle.

[0032] Both application processor 180 and image processor 190 may include various types of hardware-based processing devices. For example, either or both of application processor 180 and image processor 190 may include a microprocessor, a preprocessor (such as an image preprocessor), a graphics processor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, memory, a graphics processing unit (GPU), or any other type of device suitable for executing applications and processing and analyzing images. In some embodiments, application processor 180 and / or image processor 190 may include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, etc. Various processing devices may be used, including, for example, processors available from manufacturers such as Intel®, AMD®, or GPUs available from manufacturers such as NVIDIA®, ATI®, etc., and may include various architectures (e.g., x86 processor, ARM®, etc.).

[0033] In some embodiments, application processor 180 and / or image processor 190 may include any of the EyeQ series of processor chips available from Mobileye®. These processor designs include multiple processing units, each with its own local memory and instruction set. Such processors can include video inputs for receiving image data from multiple image sensors and may also include video output capabilities. In one example, the EyeQ2® uses 90 nm-micron technology operating at 332 MHz. The EyeQ2® architecture consists of two floating-point, hyper-threaded 32-bit RISC CPUs (MIPS32® 34K® cores), five Vision Computation Engines (VCEs), three Vector Microcode Processors (VMP®), a Denali 64-bit mobile DDR controller, a 128-bit internal Sonics Interconnect, dual 16-bit video input and 18-bit video output controllers, a 16-channel DMA, and several peripherals. The MIPS34K CPU manages five VCEs, three VMPs™ and DMA, a second MIPS34K CPU and multi-channel DMA, and other peripherals. The five VCEs, three VMPs, and the MIPS34K CPU can perform intensive vision calculations required by feature-rich bundled applications. In another example, the disclosed embodiments can use the EyeQ3™, a third-generation processor that is six times more powerful than the EyeQ2™. In another example, the disclosed embodiments can use the EyeQ4™ and / or EyeQ5™. Of course, any more recent or future EyeQ processing devices can also be used with the disclosed embodiments.

[0034] Any of the processing devices disclosed herein can be configured to perform specific functions. Configuring a processing device, such as any of the described EyeQ processors or other controllers or microprocessors, to perform specific functions may include programming computer-executable instructions and providing those instructions to the processing device for execution during operation of the processing device. In some embodiments, configuring a processing device may include directly programming architectural instructions into the processing device. For example, processing devices such as field programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs) may be configured using, for example, one or more hardware description languages ​​(HDLs).

[0035] In other embodiments, configuring the processing device may include storing executable instructions on a memory accessible by the processing device during operation. For example, the processing device may access the memory during operation to retrieve and execute the stored instructions. In either case, a processing device configured to perform the sensing, image analysis, and / or navigation functions disclosed herein represents a dedicated hardware-based system that controls multiple hardware-based components of a host vehicle.

[0036] 1 shows two separate processing devices included in processing unit 110, more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to accomplish the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by three or more processing devices. Furthermore, in some embodiments, system 100 may include one or more of processing units 110 without including other components, such as image acquisition unit 120.

[0037] The processing unit 110 may include various types of devices. For example, the processing unit 110 may include various devices such as a controller, an image preprocessor, a central processing unit (CPU), a graphics processing unit (GPU), support circuits, a digital signal processor, an integrated circuit, memory, or any other type of device for processing and analyzing images. The image preprocessor may include a video processor for capturing, digitizing, and processing images from an image sensor. The CPU may include any number of microcontrollers or microprocessors. The GPU may also include any number of microcontrollers or microprocessors. The support circuits may be any number of circuits commonly known in the art, including cache, power supplies, clocks, and input / output circuits. The memory may store software that, when executed by the processor, controls the operation of the system. The memory may include databases and image processing software. The memory may include any number of random access memories, read-only memories, flash memories, disk drives, optical storage devices, tape storage devices, removable storage devices, and other types of storage devices. In one example, the memory may be separate from the processing unit 110. In another example, the memory may be integrated into the processing unit 110.

[0038] Each memory 140, 150 may contain software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), may control the operation of various aspects of system 100. These memory units may include various databases and image processing software, as well as trained systems such as neural networks or deep neural networks. The memory units may include random access memory (RAM), read-only memory (ROM), flash memory, disk drives, optical storage devices, tape storage devices, removable storage devices, and / or any other type of storage device. In some embodiments, memory units 140, 150 may be separate from application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into application processor 180 and / or image processor 190.

[0039] Position sensor 130 may include any type of device suitable for determining a location associated with at least one component of system 100. In some embodiments, position sensor 130 may include a GPS receiver. Such a receiver may determine the location and velocity of a user by processing signals broadcast by Global Positioning System satellites. Position information from position sensor 130 may be provided to application processor 180 and / or image processor 190.

[0040] In some embodiments, system 100 may include components such as a speed sensor (e.g., a speedometer) for measuring the speed of vehicle 200. System 100 may also include one or more accelerometers (single-axis or multi-axis) for measuring the acceleration of vehicle 200 along one or more axes.

[0041] The memory units 140, 150 may contain a database or any other organized form of data that includes one or more indicators and / or locations of known landmarks. Sensor information of the environment (images from lidar or stereo processing of two or more images, radar signals, depth information, etc.) can be processed along with location information such as GPS coordinates, egomotion of the vehicle, etc. to determine the vehicle's current position relative to known landmarks and to refine the vehicle's position. Certain aspects of this technology are included in a location technology known as REM™, sold by the assignee of the present application.

[0042] User interface 170 may include any device suitable for providing information or receiving input from one or more users of system 100. In some embodiments, user interface 170 may include user input devices including, for example, a touchscreen, a microphone, a keyboard, a pointer device, a track wheel, a camera, a knob, buttons, etc. Using such input devices, a user may be able to provide information input or commands to system 100 by typing instructions or information, providing voice commands, selecting on-screen menu options using buttons, a pointer, or eye tracking, or through any other suitable technique for communicating information to system 100.

[0043] User interface 170 may comprise one or more processing devices configured to provide information to or receive information from a user and process that information, for example, for use by application processor 180. In some embodiments, such processing devices may execute instructions to recognize and track eye movements, receive and interpret voice commands, recognize and interpret touches and / or gestures made on a touchscreen, respond to keyboard entries or menu selections, etc. In some embodiments, user interface 170 may include a display, a speaker, a tactile device, and / or any other device that provides output information to a user.

[0044] Map database 160 may include any type of database that stores map data useful to system 100. In some embodiments, map database 160 may include data related to the locations in a reference coordinate system of various items, including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, etc. Map database 160 may store not only the locations of such items, but also descriptors related to those items, including, for example, names associated with any of the stored features. In some embodiments, map database 160 may be physically located with other components of system 100. Alternatively or additionally, map database 160 or portions thereof may be located remotely with respect to other components of system 100 (e.g., processing unit 110). In such embodiments, information from map database 160 may be downloaded to a network via a wired or wireless data connection (e.g., via a cellular network and / or the Internet, etc.). In some cases, map database 160 may store sparse data models including polynomial representations of specific road features (e.g., lane markings) or a desired trajectory of the host vehicle. The map database 160 may also include stored representations of various recognized landmarks that can be used to determine or update the known position of the host vehicle relative to the target trajectory. The landmark representations may include data fields such as the landmark type, the landmark location, among other potential identifiers.

[0045] Image capture devices 122, 124, and 126 may each include any type of device suitable for capturing at least one image from an environment. Furthermore, any number of image capture devices may be used to obtain images for input to the image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, four, or more image capture devices.

[0046] Further, as described above, image capture devices 122, 124, and 126 may each include any type of device suitable for capturing at least one infrared image from an environment. Any number of infrared image capture devices may be used. Some embodiments may include only a single infrared image capture device, while other embodiments may include two, three, or even four or more infrared image capture devices. Furthermore, some embodiments may include any number of infrared image capture devices in combination with any number of image capture devices. Image capture devices 122, 124, and 126 are further described below with respect to Figures 2B-2E.

[0047] The one or more cameras (e.g., image capture devices 122, 124, and 126) may be part of a sensing block included on the vehicle, which may further include one or more infrared imaging cameras, either separately or in combination with the one or more cameras.

[0048] Various other sensors may be included in the sensing block, any or all of which may be utilized to develop a sensed navigation state of the vehicle. In addition to cameras (forward, side, rear, etc.), other sensors, such as radar, lidar, acoustic sensors, etc., may be included within the sensing block. Additionally, the sensing block may include one or more components configured to communicate and transmit / receive information about the vehicle's environment. For example, such components may include a wireless transceiver (RF, etc.) that may receive sensor-based information or any other type of information related to the host vehicle's environment from a source located remotely relative to the host vehicle. Such information may include sensor output information or related information received from vehicle systems other than the host vehicle. In some embodiments, such information may include information received from a remote computing device, a centralized server, etc. Furthermore, the camera may take on many different configurations, i.e., a single camera unit, multiple cameras, a camera cluster, a long FOV, a short FOV, a wide-angle, a fisheye, etc.

[0049] System 100 or its various components may be incorporated into a variety of different platforms. In some embodiments, system 100 may be included on vehicle 200, as shown in FIG. 2A. For example, vehicle 200 may include processing unit 110 and any of the other components of system 100 described above with respect to FIG. 1. In some embodiments, vehicle 200 may include only one image capture device (e.g., a camera) and / or infrared image capture device, while in other embodiments, such as those described with respect to FIGS. 2B-2E, multiple image capture devices and / or multiple infrared capture devices may be used. For example, either of image capture devices 122 and 124 of vehicle 200 shown in FIG. 2A may be part of an ADAS (Advanced Driver Assistance Systems) imaging suite.

[0050] The image capture device included in vehicle 200 as part of image acquisition unit 120 may be located in any suitable location. In some embodiments, as shown in Figures 2A-2E and 3A-3C, image capture device 122 may be located near the rearview mirror. This location may provide a line of sight similar to that of the driver of vehicle 200 and may assist in determining what the driver can and cannot see. While image capture device 122 may be located anywhere near the rearview mirror, placing image capture device 122 on the driver's side of the mirror may further assist in capturing images representative of the driver's field of view and / or line of sight.

[0051] Other locations for the image capture devices of image acquisition unit 120 can also be used. For example, image capture device 124 can be located on or within the bumper of vehicle 200. Such a location can be particularly suitable for image capture devices with a wide field of view. The line of sight of an image capture device located on the bumper can be different from the line of sight of the driver, and thus the bumper image capture device and the driver do not always see the same object. Image capture devices (e.g., image capture devices 122, 124, and 126) can also be located in other locations. For example, image capture devices can be located on one or both side mirrors of vehicle 200, on the roof of vehicle 200, on the hood of vehicle 200, on the trunk of vehicle 200, on the sides of vehicle 200, mounted on, positioned behind, or positioned in front of any window of vehicle 200, mounted near or behind the front and / or rear lights of vehicle 200, etc.

[0052] In addition to the image capture device, vehicle 200 may include various other components of system 100. For example, processing unit 110 may be integrated into the vehicle's engine control unit (ECU) or may be included in vehicle 200 separate from the ECU. Vehicle 200 may also be equipped with a position sensor 130, such as a GPS receiver, and vehicle 200 may also include a map database 160 and memory units 140 and 150.

[0053] As described above, wireless transceiver 172 may transmit and / or receive data over one or more networks (e.g., a cellular network, the Internet, etc.). For example, wireless transceiver 172 may upload data collected by system 100 to one or more servers and download data from one or more servers. Via wireless transceiver 172, system 100 may receive updates to data stored in map database 160, memory 140, and / or memory 150, for example, periodically or on demand. Similarly, wireless transceiver 172 may upload any data from system 100 (e.g., images captured by image acquisition unit 120, data received by position sensor 130, other sensors, or vehicle control systems, etc.) and / or any data processed by processing unit 110 to one or more servers.

[0054] System 100 may upload data to a server (e.g., the cloud) based on a privacy level setting. For example, system 100 may implement a privacy level setting that regulates or limits the types of data (including metadata) that may uniquely identify the vehicle and / or the driver / owner of the vehicle that are sent to the server. Such settings may be set by a user via wireless transceiver 172, initialized by factory default settings, or set by data received by wireless transceiver 172, for example.

[0055] In some embodiments, system 100 may upload data according to a "high" privacy level, under such settings, system 100 may transmit data (e.g., location information associated with a route, captured images, etc.) without any details about the specific vehicle and / or driver / owner. For example, when uploading data according to a "high" privacy level, system 100 may not include the vehicle identification number (VIN) or the name of the vehicle's driver or owner, but instead transmit data such as captured images and / or limited location information associated with a route.

[0056] Other privacy levels are also contemplated. For example, system 100 may transmit data to a server according to a “medium” privacy level, which may include additional information not included under a “high” privacy level, such as the make and / or model of the vehicle and / or vehicle type (e.g., passenger car, sport utility vehicle, truck, etc.). In some embodiments, system 100 may upload data according to a “low” privacy level. Under the “low” privacy level setting, system 100 may upload and include sufficient data to uniquely identify a particular vehicle, owner / driver, and / or part or all of the route traveled by the vehicle. Such “low” privacy level data may include, for example, one or more of: VIN, driver / owner name, vehicle's starting point prior to departure, vehicle's intended destination, vehicle make and / or model, vehicle type, etc.

[0057] Figure 2A is a side view representation of an exemplary vehicle imaging system according to a disclosed embodiment. Figure 2B is a top view representation of the embodiment shown in Figure 2A. As shown in Figure 2B, the disclosed embodiment may show a vehicle 200 including within its body a system 100 having a first image capture device 122 positioned near a rearview mirror and / or near a driver of the vehicle 200, a second image capture device 124 positioned on or within a bumper region (e.g., one of bumper regions 210) of the vehicle 200, and a processing unit 110. In Figures 2A and 2B, one or more of the first image capture device 122 and the second image capture device 124 may include an infrared image capture device.

[0058] As shown in Figure 2C, both image capture devices 122 and 124 may be positioned near the rearview mirror and / or near the driver of vehicle 200. Similar to Figures 2A and 2B, in Figure 2C, one or more of first image capture device 122 and second image capture device 124 may include an infrared image capture device.

[0059] 2B and 2C show two image capture devices 122 and 124, it should be understood that other embodiments may include more than two image capture devices. For example, in the embodiment shown in Figures 2D and 2E, first image capture device 122, second image capture device 124, and third image capture device 126 are included in system 100 of vehicle 200. Similar to Figures 2A, 2B, and 2C, one or more of first image capture device 122, second image capture device 124, and third image capture device 126 in Figures 2D and 2E may include infrared image capture devices.

[0060] 2D , image capture device 122 may be positioned near the rearview mirror and / or near the driver of vehicle 200, and image capture devices 124 and 126 may be positioned on a bumper region (e.g., one of bumper regions 210) of vehicle 200. Also, as shown in FIG. 2E , image capture devices 122, 124, and 126 may be positioned near the rearview mirror and / or near the driver's seat of vehicle 200. The disclosed embodiments are not limited to any particular number or configuration of image capture devices, and image capture devices may be positioned in any suitable location within and / or on vehicle 200.

[0061] It should be understood that the disclosed embodiments are not limited to vehicles and may be applicable in other contexts. It should also be understood that the disclosed embodiments are not limited to a particular type of vehicle 200 and may be applicable to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.

[0062] First image capture device 122 may include any suitable type of image capture device or infrared image capture device. Image capture device 122 may include an optical axis. In one example, image capture device 122 may include an Aptina M9V024W VGA sensor with a global shutter. In other embodiments, image capture device 122 may provide a resolution of 1280 x 960 pixels and may include a rolling shutter. Image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included to provide, for example, a desired focal length and field of view for the image capture device. In some embodiments, image capture device 122 may be associated with a 6 mm lens or a 12 mm lens. In some embodiments, image capture device 122 may be configured to capture an image having a desired field of view (FOV) 202, as shown in FIG. 2D . For example, image capture device 122 may be configured to have a conventional FOV, such as within the range of 40 degrees to 56 degrees, including a 46 degree FOV, a 50 degree FOV, a 52 degree FOV, or degrees greater than 52 degrees. Alternatively, image capture device 122 may be configured to have a narrower FOV, such as a 23-40 degree FOV, such as a 28 degree FOV or a 36 degree FOV. Additionally, image capture device 122 may be configured to have a wider FOV, such as a 100-180 degree FOV. In some embodiments, image capture device 122 may include a wide-angle bumper camera or a bumper camera with an FOV of up to 180 degrees. In some embodiments, image capture device 122 may be a 7.2 Mpixel image capture device with an aspect ratio of approximately 2:1 (e.g., H×V=3800×1900 pixels) with a horizontal FOV of approximately 100 degrees. Such an image capture device may be used instead of a three-dimensional image capture device configuration. Due to large lens distortion, the vertical FOV of such image capture devices can be much less than 50 degrees in implementations in which the image capture device uses a radially symmetric lens, for example, such lenses are not radially symmetric, thereby allowing a vertical FOV greater than 50 degrees with a horizontal FOV of 100 degrees.

[0063] First image capture device 122 may acquire a plurality of first images of a scene associated with vehicle 200. Each of the plurality of first images may be acquired as a series of image scan lines, which may be captured using a rolling shutter. Each scan line may include a plurality of pixels. In embodiments in which first image capture device 122 includes an infrared image capture device, each of the plurality of first images may be acquired as a series of image scan lines, which may be captured using an electronic scanning system.

[0064] First image capture device 122 may have a scan rate associated with acquiring each of the first series of image scan lines. The scan rate may refer to the rate at which the image sensor can acquire image data associated with each pixel included in a particular scan line. In embodiments in which first image capture device 122 includes an infrared image capture device, the scan rate may refer to the rate at which the infrared image sensor can acquire thermal data associated with each pixel included in a particular scan line.

[0065] Image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, including, for example, CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be utilized with a rolling shutter, whereby each pixel in a row is read one at a time, and the scanning of the rows is advanced row by row until the entire image frame is captured. In some embodiments, the rows may be captured sequentially from top to bottom relative to the frame. In embodiments in which one or more of image capture devices 122, 124, and 126 include infrared image capture devices, an uncooled focal plane array (UFPA) may be utilized with an electronic scanning system, whereby the scanning of the rows is advanced row by row until the entire thermal map is captured.

[0066] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124, and 126) may constitute high-resolution imagers and may have a resolution of greater than 5M pixels, greater than 7M pixels, greater than 10M pixels, or more.

[0067] Using a rolling shutter can result in pixels in different rows being exposed and captured at different times, which can cause skew and other image artifacts in the captured image frame. On the other hand, if image capture device 122 is configured to operate with a global or synchronous shutter, all of the pixels can be exposed for the same amount of time and during a common exposure period. As a result, image data in a frame collected from a system utilizing a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a particular point in time. In contrast, in a rolling shutter application, each row in a frame is exposed at a different time, and data is captured at a different time. Therefore, moving objects can appear distorted in an image capture device with a rolling shutter. This phenomenon (which equally applies to the use of electronic scanning in infrared image capture devices) is described in more detail below.

[0068] Second image capture device 124 and third image capture device 126 may be any type of image capture device or infrared image capture device. Like first image capture device 122, each of image capture devices 124 and 126 may include an optical axis. In one embodiment, each of image capture devices 124 and 126 may include an Aptina M9V024W VGA sensor with a global shutter. Alternatively, each of image capture devices 124 and 126 may include a rolling shutter. Like image capture device 122, image capture devices 124 and 126 may be configured to include various lenses and optical elements. In some embodiments, the lenses associated with image capture devices 124 and 126 may provide the same FOV as that associated with image capture device 122 (such as FOV 202) or a narrower FOV (such as FOVs 204 and 206). For example, image capture devices 124 and 126 may have an FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less than 20 degrees.

[0069] Image capture devices 124 and 126 may acquire multiple second and third images of a scene associated with vehicle 200. Each of the multiple second and third images may be acquired as a second and third series of image scan lines, which may be captured using a rolling shutter. Each scan line or row may have a plurality of pixels. Image capture devices 124 and 126 may have second and third scan rates associated with acquiring each image scan line included in the second and third series. In embodiments in which one or more of image capture devices 124 and 126 include infrared image capture devices, each of the multiple second and third images may be acquired as a series of second and third thermal scan lines, which may be captured using an electronic scanning system. In such embodiments, each scan line or row may have a plurality of pixels, and image capture devices 124 and / or 126 may have second and third scan rates associated with acquiring each thermal scan line included in the second and third series.

[0070] Each image capture device 122, 124, and 126 may be positioned in any suitable location and in any suitable orientation relative to vehicle 200. The relative positions of image capture devices 122, 124, and 126 may be selected to facilitate fusing together information obtained from the image capture devices. For example, in some embodiments, the FOV associated with image capture device 124 (FOV 204) may partially or completely overlap with the FOV associated with image capture device 122 (e.g., FOV 202) and the FOV associated with image capture device 126 (e.g., FOV 206).

[0071] Image capture devices 122, 124, and 126 may be positioned on vehicle 200 at any suitable relative height. In one example, there may be a height difference between image capture devices 122, 124, and 126, which may provide sufficient parallax information to enable stereo analysis. For example, as shown in FIG. 2A , two image capture devices 122 and 124 are at different heights. There may also be a lateral displacement difference between image capture devices 122, 124, and 126, which may provide additional parallax information for stereo analysis by processing unit 110, for example. The lateral displacement difference may be denoted by dx, as shown in FIGS. 2C and 2D . In some embodiments, a front displacement or a rear displacement (e.g., a range displacement) may exist between image capture devices 122, 124, and 126. For example, image capture device 122 may be positioned 0.5 to 2 meters or more behind image capture device 124 and / or image capture device 126. This type of displacement may allow one of the image capture devices to cover a potential blind spot of the other image capture device.

[0072] Similarly, there may be no elevation difference between image capture devices 122, 124, and 126, which may assist in aligning a thermal map created by one or more of the image capture devices with a visible light image created by one or more of the image capture devices.

[0073] Image capture device 122 may have any suitable resolution capability (e.g., number of pixels associated with the image sensor), and the resolution of the image sensor associated with image capture device 122 may be higher, lower, or the same as the resolution of the image sensors associated with image capture devices 124 and 126. In some embodiments, the image sensors associated with image capture device 122 and / or image capture devices 124 and 126 may have a resolution of 640x480, 1024x768, 1280x960, or any other suitable resolution.

[0074] The frame rate (e.g., the rate at which the image capture device acquires pixel data or thermal data sets for one image frame before moving on to capture pixel data or thermal data associated with the next image frame) may be controllable. The frame rate associated with image capture device 122 may be higher, lower, or the same as the frame rate associated with image capture devices 124 and 126. The frame rates associated with image capture devices 122, 124, and 126 may depend on various factors that may affect the timing of the frame rate. For example, one or more of image capture devices 122, 124, and 126 may include a selectable pixel delay period imposed before or after acquisition of image data associated with one or more pixels of an image sensor within image capture devices 122, 124, and / or 126. Generally, image data corresponding to each pixel may be acquired according to the device's clock rate (e.g., one pixel per clock cycle). Additionally, in embodiments including a rolling shutter, one or more of image capture devices 122, 124, and 126 may include a selectable horizontal blanking period imposed before or after acquisition of image data associated with a row of pixels of an image sensor in image capture devices 122, 124, and / or 126. Additionally, one or more of image capture devices 122, 124, and / or 126 may include a selectable vertical blanking period imposed before or after acquisition of image data associated with an image frame of image capture devices 122, 124, and 126. Similarly, in embodiments including electronic scanning, one or more of image capture devices 122, 124, and 126 may include a dynamically variable scan rate.

[0075] These timing controls may enable the synchronization of frame rates associated with image capture devices 122, 124, and 126, even if the line scan rate of each image capture device is different. Additionally, as discussed in more detail below, among other factors (e.g., image sensor resolution, maximum line scan rate, etc.), these selectable timing controls may enable the synchronization of image capture from areas where the FOV of image capture device 122 overlaps with one or more FOVs of image capture devices 124 and 126, even if the field of view of image capture device 122 differs from the FOVs of image capture devices 124 and 126.

[0076] The frame rate timing for image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensors. For example, assuming both devices have similar line scan rates, if one device includes an image sensor with a resolution of 640x480 and the other device includes an image sensor with a resolution of 1280x960, it will take longer to capture a frame of image data from the sensor with the higher resolution.

[0077] Another factor that may affect the timing of image data acquisition at image capture devices 122, 124, and 126 is the maximum line scan rate. For example, acquisition of a row of image data from the image sensors included in image capture devices 122, 124, and 126 requires some minimum amount of time. Assuming no pixel delay period is added, this minimum amount of time to acquire a row of image data will be related to the maximum line scan rate of a particular device. Devices offering higher maximum line scan rates have the potential to provide higher frame rates than devices with lower maximum line scan rates. In some embodiments, one or both of image capture devices 124 and 126 may have a maximum line scan rate that is higher than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture devices 124 and / or 126 may be 1.25, 1.5, 1.75, or 2 or more times the maximum line scan rate of image capture device 122.

[0078] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may operate at a scan rate that is equal to or less than that maximum scan rate. The system may be configured so that one or both of image capture devices 124 and 126 operate at a line scan rate that is equal to the line scan rate of image capture device 122. In other examples, the system may be configured so that the line scan rate of image capture devices 124 and / or 126 may be 1.25, 1.5, 1.75, or 2 or more times the line scan rate of image capture device 122.

[0079] In some embodiments, image capture devices 122, 124, and 126 may be asymmetric. That is, the image capture devices may include cameras with different fields of view (FOV) and focal lengths. The fields of view of image capture devices 122, 124, and 126 may include, for example, any desired area of ​​the environment of vehicle 200. In some embodiments, one or more of image capture devices 122, 124, and 126 may be configured to acquire image data from the environment in front of vehicle 200, the environment behind vehicle 200, the environment on either side of vehicle 200, or a combination thereof.

[0080] Additionally, the focal length associated with each image capture device 122, 124, and / or 126 may be selectable (e.g., by inclusion of an appropriate lens, etc.) so that each device captures images of objects at a desired distance range from vehicle 200. For example, in some embodiments, image capture devices 122, 124, and 126 may capture images of close-up objects within a few meters of the vehicle. Image capture devices 122, 124, 126 may also be configured to capture images of objects at greater distances from the vehicle (e.g., 25 m, 50 m, 100 m, 150 m, or more). Furthermore, the focal lengths of image capture devices 122, 124, and 126 may be selected such that one image capture device (e.g., image capture device 122) can capture images of objects relatively close to the vehicle (e.g., within 10 m or within 20 m), while the other image capture devices (e.g., image capture devices 124 and 126) can capture images of objects farther away from vehicle 200 (e.g., more than 20 m, more than 50 m, more than 100 m, more than 150 m, etc.).

[0081] According to some embodiments, the FOV of one or more of image capture devices 122, 124, and 126 may have a wide angle. For example, it may be advantageous to have an FOV of 140 degrees, particularly for image capture devices 122, 124, and 126 that may be used to capture images of areas near vehicle 200. For example, image capture device 122 may be used to capture images of areas to the right or left of vehicle 200, and in such embodiments, it may be desirable for image capture device 122 to have a wide FOV (e.g., at least 140 degrees).

[0082] The field of view associated with each of image capture devices 122, 124, and 126 may depend on the respective focal lengths. For example, as the focal lengths increase, the corresponding field of view decreases.

[0083] Image capture devices 122, 124, and 126 may be configured to have any suitable field of view. In one particular example, image capture device 122 may have a horizontal FOV of 46 degrees, image capture device 124 may have a horizontal FOV of 23 degrees, and image capture device 126 may have a horizontal FOV between 23 and 46 degrees. In another example, image capture device 122 may have a horizontal FOV of 52 degrees, image capture device 124 may have a horizontal FOV of 26 degrees, and image capture device 126 may have a horizontal FOV between 26 and 52 degrees. In some embodiments, the ratio between the FOV of image capture device 122 and the FOV of image capture device 124 and / or image capture device 126 may vary between 1.5 and 2.0. In other embodiments, this ratio may vary between 1.25 and 2.25.

[0084] System 100 may be configured so that the field of view of image capture device 122 at least partially or completely overlaps the field of view of image capture device 124 and / or image capture device 126. In some embodiments, system 100 may be configured so that the fields of view of image capture devices 124 and 126, for example, fall within the field of view of image capture device 122 (e.g., are smaller than the field of view of image capture device 122) and share a common center with the field of view of image capture device 122. In other embodiments, image capture devices 122, 124, and 126 may capture adjacent FOVs or may have partially overlapping FOVs. In some embodiments, the fields of view of image capture devices 122, 124, and 126 may be aligned such that the center of image capture device 124 and / or 126, with the narrower FOV, may be located in the bottom half of the field of view of device 122, with the wider FOV.

[0085] 2F is a diagrammatic representation of an exemplary vehicle control system according to disclosed embodiments. As shown in FIG. 2F, vehicle 200 may include a throttle system 220, a braking system 230, and a steering system 240. System 100 may provide inputs (e.g., control signals) to one or more of throttle system 220, braking system 230, and steering system 240 via one or more data links (e.g., any wired and / or wireless link or link that transmits data). For example, based on analysis of images acquired by image capture devices 122, 124, and / or 126, system 100 may provide control signals to one or more of throttle system 220, braking system 230, and steering system 240 to navigate vehicle 200 (e.g., by causing it to accelerate, turn, shift lanes, etc.). Additionally, system 100 may receive inputs indicative of the operating conditions of vehicle 200 (e.g., speed, whether vehicle 200 is braking and / or turning, etc.) from one or more of throttle system 220, braking system 230, and steering system 24. Further details are provided below in connection with Figures 4-7.

[0086] 3A , vehicle 200 may also include a user interface 170 for interacting with a driver or passenger of vehicle 200. For example, user interface 170 in a vehicle application may include a touchscreen 320, knobs 330, buttons 340, and a microphone 350. A driver or passenger of vehicle 200 may also interact with system 100 using a steering wheel (e.g., including a turn signal handle and located on or near the steering column of vehicle 200), buttons (e.g., located on the steering wheel of vehicle 200), and the like. In some embodiments, microphone 350 may be positioned adjacent to rearview mirror 310. Similarly, in some embodiments, image capture device 122 may be located near rearview mirror 310. In some embodiments, user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, system 100 may provide various notifications (e.g., alerts) via speaker 360.

[0087] 3B-3D are diagrams of an exemplary camera mount 370 configured to be positioned behind a rearview mirror (e.g., rearview mirror 310) and opposite a vehicle windshield, according to disclosed embodiments. As shown in FIG. 3B, camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be positioned behind glare shield 380, which may be in direct contact with the windshield and may include a film and / or anti-reflective material composition. For example, glare shield 380 may be positioned to be aligned opposite a windshield having a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 may be positioned behind glare shield 380, for example, as shown in FIG. 3D. In embodiments in which one or more of image capture devices 122, 124, and 126 include infrared image capture devices, these devices may be positioned in front of glareshield 380 to prevent such materials from interfering with the capture of infrared light (or alternatively, glareshield 380 may not extend in front of the infrared image capture devices). The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124, and 126, camera mount 370, and glareshield 380. Figure 3C is a view of camera mount 370 shown in Figure 3B from the front.

[0088] As will be appreciated by those skilled in the art having the benefit of this disclosure, many variations and / or modifications may be made to the above-disclosed embodiments. For example, not all components are essential to the operation of system 100. Furthermore, any component may be located in any suitable portion of system 100, and the components may be rearranged in various configurations while still providing the functionality of the disclosed embodiments. Accordingly, the above-described configurations are examples, and regardless of the above-described configuration, system 100 may provide a wide range of functionality for analyzing the surroundings of vehicle 200 and navigating vehicle 200 in response to the analysis.

[0089] As discussed in more detail below, according to various disclosed embodiments, system 100 may provide various features related to autonomous driving and / or driver assistance technologies. For example, system 100 may analyze image data, infrared image data, location data (e.g., GPS location information), map data, speed data, and / or data from sensors included in vehicle 200. System 100 may collect data for analysis from, for example, image acquisition unit 120, location sensor 130, and other sensors. Furthermore, system 100 may analyze the collected data to identify whether vehicle 200 should take a particular action and then automatically take the determined action without human intervention. For example, when vehicle 200 navigates without human participation, system 100 may automatically control the braking, acceleration, and / or steering of vehicle 200 (e.g., by sending control signals to one or more of throttle system 220, braking system 230, and steering system 240). Additionally, system 100 may analyze the collected data and issue warnings and / or alerts to vehicle occupants based on the analysis of the collected data. Further details regarding various embodiments provided by system 100 are provided below.

[0090] Forward-Facing Multi-Imaging System As described above, system 100 may provide driver assistance features using a multi-camera system. The multi-camera system may use one or more cameras (and / or infrared cameras) facing forward of the vehicle. In other embodiments, the multi-camera system may include one or more cameras (and / or infrared cameras) facing toward the side of the vehicle or toward the rear of the vehicle. In one embodiment, for example, system 100 may use a two-camera imaging system, in which a first camera and a second camera (e.g., image capture devices 122 and 124) may be located at the front and / or side of the vehicle (e.g., vehicle 200). Other camera configurations are consistent with the disclosed embodiments, and the configurations disclosed herein are examples. For example, system 100 may include any number (e.g., 1, 2, 3, 4, 5, 6, 7, 8, etc.) of camera configurations and any combination of camera types (e.g., two visual cameras and one infrared camera, one visual camera and two infrared cameras, two visual cameras and two infrared cameras, etc.). Additionally, system 100 may include "clusters" of cameras. For example, a cluster of cameras (including any suitable number (e.g., 1, 4, 8, etc.) of cameras and any suitable type (e.g., visual cameras, infrared cameras, etc.)) may face forward relative to the vehicle or may face in any other direction (e.g., facing forward, sideways, diagonally, etc.). Thus, system 100 may include multiple clusters of cameras, each oriented in a particular direction to capture images from a particular region of the vehicle's environment.

[0091] The first camera may have a field of view that is larger than, smaller than, or partially overlaps with, the field of view of the second camera. Additionally, the first camera may be connected to a first image processor to perform monocular image analysis of images provided by the first camera, and the second camera may be connected to a second image processor to perform monocular image analysis of images provided by the second camera. In embodiments in which one or more of the first and second cameras include an infrared camera, the first image processor and / or the second image processor may perform thermal map analysis of the thermal map provided by the infrared camera.

[0092] The outputs (e.g., processed information) of the first and second image processors may be combined. In some embodiments, the second image processor may receive images from both the first and second cameras and perform stereo analysis, or perform analysis on aligned visible light and infrared images. In another embodiment, system 100 may use a three-camera imaging system, where each camera has a different field of view. Thus, such a system may make decisions based on information derived from objects at various distances both in front of and to the sides of the vehicle. References to monocular image analysis may refer to when image analysis is performed based on images captured from a single viewpoint (e.g., from a single camera). Stereo image analysis may refer to when image analysis is performed based on two or more images captured with one or more image capture parameters changed. For example, captured images suitable for performing stereo image analysis may include images captured from two or more different positions, images captured from different fields of view, images captured using different focal lengths, images captured with parallax information, etc. Hybrid image analysis may refer to when one or more visible light images are registered with one or more infrared images and image analysis is performed based on the registered images.

[0093] For example, in one embodiment, system 100 may implement a three-camera configuration using image capture devices 122, 124, and 126. In such a configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees or other value selected from the range of approximately 20 to 45 degrees), image capture device 124 may provide a wide field of view (e.g., 150 degrees or other value selected from the range of approximately 100 to approximately 180 degrees), and image capture device 126 may provide a medium field of view (e.g., 46 degrees or other value selected from the range of approximately 35 to approximately 60 degrees). In some embodiments, image capture device 126 may operate as the main or primary camera. Image capture devices 122, 124, and 126 may be positioned substantially side-by-side (e.g., 6 cm apart) behind rearview mirror 310. Additionally, in some embodiments, as described above, one or more of image capture devices 122, 124, and 126 may be mounted behind a glare shield 380 that is flush with the windshield of vehicle 200. Such a shield may operate to minimize the impact of any reflections from the interior of the vehicle on image capture devices 122, 124, and 126.

[0094] 3B and 3C, the wide field of view camera (e.g., image capture device 124 in the example above) may be mounted lower than the narrow main field of view camera (e.g., image capture devices 122 and 126 in the example above). This configuration may provide a free line of sight from the wide field of view camera. To reduce reflections, the camera may be mounted near the windshield of vehicle 200 and may include a polarizer to attenuate reflected light.

[0095] A three-camera system may offer certain performance features. For example, some embodiments may include the ability to verify the detection of an object by one camera based on the detection results from another camera. In the three-camera configuration described above, processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips as described above), each dedicated to processing images captured by one or more of image capture devices 122, 124, and 126.

[0096] In a three-camera system, the first processing device may receive images from both the primary camera and the narrow FOV camera and perform vision processing for the narrow FOV camera to detect, for example, other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the first processing device may calculate pixel disparity between images from the primary camera and the narrow camera to create a 3D reconstruction of the environment of vehicle 200. The first processing device may then combine the 3D reconstruction with 3D map data or 3D information calculated based on information from another camera.

[0097] The second processing device may receive images from the primary camera and perform vision processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Furthermore, the second processing device may calculate camera displacement, calculate pixel disparity between successive images based on the displacement, and create 3D reconstruction information (e.g., structure-from-motion) of the scene. The second processing device may send the structure-from-motion based on the 3D reconstruction information to the first processing device and combine the structure-from-motion with the stereoscopic 3D image.

[0098] The third processing device may receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device may further execute additional processing instructions to analyze the images and identify moving objects in the images, such as vehicles changing lanes, pedestrians, etc.

[0099] In some embodiments, having image-based information streams captured and processed independently may provide an opportunity for redundancy in the system, such as using a first image capture device and images processed from that device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.

[0100] In some embodiments, system 100 may use two image capture devices (e.g., image capture devices 122 and 124) in providing navigation assistance to vehicle 200, and may use a third image capture device (e.g., image capture device 126) to provide redundancy and verify the analysis of data received from the other two image capture devices. For example, in such a configuration, image capture devices 122 and 124 may provide images for stereo analysis by system 100 for navigating vehicle 200, while image capture device 126 may provide images for monocular analysis by system 100 to provide redundancy and verification of information derived based on images captured from image capture device 122 and / or image capture device 124. That is, image capture device 126 (and corresponding processing device) may be considered to provide a redundant subsystem that provides a check on the analysis derived from image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AEB) system). Additionally, in some embodiments, redundancy and validation of received data can be supplemented based on information received from one or more sensors (e.g., radar, lidar, acoustic sensors, information received from one or more transceivers outside the vehicle, etc.).

[0101] Those skilled in the art will recognize that the above camera configurations, camera placements, camera numbers, camera locations, etc. are merely exemplary. These components, etc., described for an overall system, may be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of multi-camera systems to provide driver assistance and / or autonomous vehicle functionality follow below.

[0102] 4 is an example functional block diagram of memory 140 and / or 150 that may be stored / programmed with instructions to perform one or more operations in accordance with the disclosed embodiments. While reference is made below to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0103] 4 , memory 140 may store a monocular image analysis module 402, a stereo image analysis module 404, a velocity and acceleration module 406, and a navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 402, 404, 406, and 408 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Accordingly, any steps of the following processes may be performed by one or more processing devices.

[0104] In one embodiment, monocular image analysis module 402 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, perform monocular image analysis of a set of images acquired by one of image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from the set of images with additional sensor information (e.g., information from radar, lidar, etc.) to perform the monocular image analysis. As described in connection with FIGS. 5A-5D below, monocular image analysis module 402 may include instructions to detect a set of features in the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazards, and any other features associated with the vehicle's environment. Based on the analysis, system 100 (e.g., via processing unit 110) may cause one or more navigational responses in vehicle 200, such as a turn, a lane shift, and an acceleration change, as discussed below in connection with navigation response module 408.

[0105] In one embodiment, stereo image analysis module 404 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, perform stereo image analysis of first and second sets of images acquired by a combination of image capture devices selected from image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from the first and second sets of images with additional sensor information (e.g., information from radar) to perform stereo image analysis. For example, stereo image analysis module 404 may include instructions to perform stereo image analysis based on the first set of images acquired by image capture device 124 and the second set of images acquired by image capture device 126. As described below in connection with FIG. 6 , stereo image analysis module 404 may include instructions to detect a set of features in the first and second sets of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and hazards. Based on the analysis, processing unit 110 may cause one or more navigation responses in vehicle 200, such as turns, lane shifts, and acceleration changes, as described below in connection with navigation response module 408. Additionally, in some embodiments, stereo image analysis module 404 may implement techniques related to trained systems (such as neural networks or deep neural networks) or untrained systems (such as systems that may be configured to use computer vision algorithms) to detect and / or label objects in the environment from which the sensor information was captured and processed. In one embodiment, stereo image analysis module 404 and / or other image processing modules may be configured to use a combination of trained and untrained systems.

[0106] In one embodiment, speed and acceleration module 406 may store software configured to analyze data received from one or more computational and electromechanical devices within vehicle 200, configured to alter the speed and / or acceleration of vehicle 200. For example, processing unit 110 may execute instructions associated with speed and acceleration module 406 to calculate a target speed of vehicle 200 based on data derived from the execution of monocular image analysis module 402 and / or stereo image analysis module 404. Such data may include, for example, target position, speed, and / or acceleration, the position and / or speed of vehicle 200 relative to nearby vehicles, pedestrians, or road objects, and position information of vehicle 200 relative to lane markings on the road. Additionally, processing unit 110 may calculate a target speed of vehicle 200 based on sensor input (e.g., information from radar) and input from other systems of vehicle 200, such as throttle system 220, braking system 230, and / or steering system 240 of vehicle 200. Based on the calculated target speed, the processing unit 110 may send electronic signals to the throttle system 220, the braking system 230, and / or the steering system 240 of the vehicle 200 to trigger a change in speed and / or acceleration, for example, by physically releasing the brakes or easing the accelerator of the vehicle 200.

[0107] In one embodiment, the navigation response module 408 may be executable by the processing unit 110 and may store software that determines a desired navigation response based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include position and speed information associated with nearby vehicles, pedestrians, and road objects, as well as target position information for the vehicle 200. Furthermore, in some embodiments, the navigation response may be based (partially or fully) on map data, a predetermined position of the vehicle 200, and / or a relative speed or relative acceleration between the vehicle 200 and one or more objects detected from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 may also determine a desired navigation response based on sensor inputs (e.g., information from radar) and inputs from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and the steering system 240 of the vehicle 200. Based on the desired navigation response, processing unit 110 may send electronic signals to throttle system 220, braking system 230, and steering system 240 of vehicle 200 to trigger the desired navigation response, for example, by turning the steering wheel of vehicle 200 to achieve a predetermined angle of rotation. In some embodiments, processing unit 110 may use the output of navigation response module 408 (e.g., the desired navigation response) as input to the execution of velocity and acceleration module 406 to calculate a change in velocity of vehicle 200.

[0108] Additionally, any of the modules disclosed herein (e.g., modules 402, 404, and 406) may implement techniques related to trained systems (such as neural networks or deep neural networks) or untrained systems.

[0109] 5A is a flowchart illustrating an example process 500A for generating one or more navigational responses based on monocular image analysis, according to a disclosed embodiment. At step 510, processing unit 110 may receive a plurality of images via data interface 128 between processing unit 110 and image acquisition unit 120. For example, a camera (such as image capture device 122 having field of view 202) included in image acquisition unit 120 may capture a plurality of images of an area in front of vehicle 200 (or, for example, to the side or rear of the vehicle) and transmit them to processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). Processing unit 110 may execute monocular image analysis module 402 to analyze the plurality of images at step 520, as described in further detail below in connection with FIGS. 5B-5D . By performing the analysis, processing unit 110 may detect a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, and traffic lights.

[0110] Processing unit 110 may also execute monocular image analysis module 402 in step 520 to detect various road hazards, such as truck tire parts, fallen road signs, loose cargo, and small animals. Road hazards may vary in structure, shape, size, and color, making such hazards more difficult to detect. In some embodiments, processing unit 110 may execute monocular image analysis module 402 and perform multi-frame analysis on multiple images to detect road hazards. For example, processing unit 110 may estimate camera movement between consecutive image frames and calculate pixel disparity between frames to construct a 3D map of the road. Processing unit 110 may then use the 3D map to detect the road surface and hazards present on the road surface.

[0111] In step 530, processing unit 110 may execute navigation response module 408 to cause vehicle 200 to make one or more navigational responses based on the analysis performed in step 520 and the techniques described above in connection with FIG. 4 . Navigational responses may include, for example, a turn, a lane shift, an acceleration change, etc. In some embodiments, processing unit 110 may use data derived from execution of speed and acceleration module 406 to cause one or more navigational responses. Furthermore, multiple navigational responses may occur simultaneously, sequentially, or any combination thereof. For example, processing unit 110 may cause vehicle 200 to shift one lane and then, for example, accelerate, for example, by sequentially sending control signals to steering system 240 and throttle system 220 of vehicle 200. Alternatively, processing unit 110 may cause vehicle 200 to brake and simultaneously shift lanes, for example, by simultaneously sending control signals to braking system 230 and steering system 240 of vehicle 200.

[0112] FIG. 5B is a flowchart illustrating an example process 500B for detecting one or more vehicles and / or pedestrians in a set of images according to a disclosed embodiment. Processing unit 110 may execute monocular image analysis module 402 to perform process 500B. In step 540, processing unit 110 may identify a set of candidate objects representing possible vehicles and / or pedestrians. For example, processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify locations within each image that may contain a target object (e.g., a vehicle, a pedestrian, or portions thereof). The predetermined patterns may be specified to achieve a low rate of “false hits” and a low rate of “misses.” For example, processing unit 110 may use a low similarity threshold to the predetermined patterns to identify candidate objects as possible vehicles or pedestrians. By doing so, processing unit 110 may reduce the probability of missing (e.g., not identifying) a candidate object representing a vehicle or pedestrian.

[0113] At step 542, processing unit 110 may filter the set of candidate objects to eliminate certain candidates (e.g., irrelevant or less relevant objects) based on classification criteria. Such criteria may be derived from various characteristics associated with object types stored in a database (e.g., a database stored in memory 140). The characteristics may include the object's shape, dimensions, texture, and location (e.g., relative to vehicle 200), etc. Thus, processing unit 110 may use one or more sets of criteria to reject false candidates from the set of candidate objects.

[0114] At step 544, processing unit 110 may analyze multiple image frames to identify whether objects in the set of candidate images represent vehicles and / or pedestrians. For example, processing unit 110 may track detected candidate objects across successive frames and accumulate frame-by-frame data associated with the detected objects (e.g., size, position relative to vehicle 200, etc.). Additionally, processing unit 110 may estimate parameters of the detected objects and compare the object's frame-by-frame position data to predicted positions.

[0115] At step 546, processing unit 110 may construct a set of measurements of the detected objects. Such measurements may include, for example, position, velocity, and acceleration values ​​(e.g., relative to vehicle 200) associated with the detected objects. In some embodiments, processing unit 110 may construct the measurements based on estimation techniques that use a series of time-based observations, such as a Kalman filter or linear quadratic estimation (LQE), and / or modeling data available for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). A Kalman filter may be based on measurements of the scale of the objects, where the scale measurement is proportional to the time to impact (e.g., the amount of time it takes for vehicle 200 to reach the object). Thus, by performing steps 540, 542, 544, and 546, processing unit 110 may identify vehicles and pedestrians appearing in the set of captured images and derive information (e.g., position, velocity, size) associated with the vehicles and pedestrians. Based on the identified and derived information, processing unit 110 may cause one or more navigation responses in vehicle 200, as described above in connection with FIG. 5A.

[0116] In step 548, processing unit 110 may perform optical flow analysis of one or more images to reduce the probability of detecting “false hits” and missing candidate objects representing vehicles or pedestrians. Optical flow analysis may refer to analyzing movement patterns, separate from road surface movement, for vehicle 200 in one or more images associated with other vehicles and pedestrians, for example. Processing unit 110 may calculate the movement of candidate objects by observing different positions of the object across multiple image frames captured at different times. Processing unit 110 may use the position and time values ​​as inputs to a mathematical model to calculate the movement of candidate objects. Thus, optical flow analysis may provide another method of detecting vehicles and pedestrians in the vicinity of vehicle 200. Processing unit 110 may perform optical flow analysis in combination with steps 540, 542, 544, and 546 to provide redundancy for detecting vehicles and pedestrians and increase the reliability of system 100.

[0117] 5C is a flowchart illustrating an example process 500C for detecting road marks and / or lane geometry information in a set of images according to the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500C. In step 550, processing unit 110 may detect a set of objects by scanning one or more images. To detect lane mark segments, lane geometry information, and other related road marks, processing unit 110 may filter the set of objects to exclude those determined to be irrelevant (e.g., small holes, small rocks, etc.). In step 552, processing unit 110 may group together segments detected in step 550 that belong to the same road or lane mark. Based on the grouping, processing unit 110 may develop a model, such as a mathematical model, to represent the detected segments.

[0118] At step 554, processing unit 110 may construct a set of measurements associated with the detected segment. In some embodiments, processing unit 110 may create a projection of the detected segment from the image plane onto the real-world plane. The projection may be characterized using a third-order polynomial with coefficients corresponding to physical properties of the detected road, such as its position, slope, curvature, and curvature derivative. In generating the projection, processing unit 110 may take into account road surface variations and pitch and roll rates associated with vehicle 200. Additionally, processing unit 110 may model road height by analyzing motion cues present in the position and road surface. Furthermore, processing unit 110 may estimate pitch and roll rates associated with vehicle 200 by tracking a set of feature points in one or more images.

[0119] In step 556, processing unit 110 may perform a multi-frame analysis, for example, by tracking the detected segment across successive image frames and accumulating frame-by-frame data associated with the detected segment. When processing unit 110 performs the multi-frame analysis, the set of measurements constructed in step 554 may become more reliable and may be associated with an increasingly higher degree of confidence. Thus, by performing steps 550-556, processing unit 110 may identify road marks appearing in the set of captured images and derive lane geometry information. Based on the identification and derived information, processing unit 110 may cause one or more navigational responses in vehicle 200, as described above in connection with FIG. 5A .

[0120] In step 558, processing unit 110 may consider additional information sources to further develop a safety model of vehicle 200 in the vehicle's surroundings. Processing unit 110 may use the safety model to define situations in which system 100 may safely perform autonomous control of vehicle 200. To develop the safety model, in some embodiments, processing unit 110 may consider the positions and movements of other vehicles, detected road edges and barriers, and / or general road shape descriptions extracted from map data (such as data from map database 160). By considering additional information sources, processing unit 110 may provide redundancy in detecting road marks and lane geometry and increase the reliability of system 100.

[0121] FIG. 5D is a flowchart illustrating an example process 500D for detecting traffic lights in a set of images according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500D. In step 560, processing unit 110 may scan the set of images and identify objects that appear at locations in the images that are likely to contain traffic lights. For example, processing unit 110 may filter the identified objects to construct a set of candidate objects that excludes objects that are unlikely to correspond to traffic lights. The filtering may be based on various characteristics associated with traffic lights, such as shape, size, texture, and location (e.g., relative to vehicle 200). Such characteristics may be based on many examples of traffic lights and traffic control signals and may be stored in a database. In some embodiments, processing unit 110 may perform multi-frame analysis on the set of candidate objects that reflect possible traffic lights. For example, processing unit 110 may track the candidate objects across consecutive image frames, estimate the real-world locations of the candidate objects, and filter out moving objects (which are unlikely to be traffic lights). In some embodiments, processing unit 110 may perform color analysis on the candidate object to identify the relative location of the detected color represented within the potential traffic light.

[0122] In step 562, processing unit 110 may analyze the geometry of the intersection. The analysis may be based on any combination of (i) the number of lanes detected on either side of vehicle 200, (ii) marks (such as arrow marks) detected on the road, and (iii) a description of the intersection extracted from map data (such as data from map database 160). Processing unit 110 may perform the analysis using information derived from execution of monocular analysis module 402. In addition, processing unit 110 may identify correspondences between traffic lights detected in step 560 and lanes appearing near vehicle 200.

[0123] As vehicle 200 approaches the intersection, in step 564, processing unit 110 may update the confidence associated with the analyzed intersection geometry and detected traffic lights. For example, the number of traffic lights estimated to appear at the intersection compared to the number of traffic lights that actually appear at the intersection may affect the confidence. Thus, based on the confidence, processing unit 110 may delegate control to the driver of vehicle 200 to improve the safety situation. By performing steps 560, 562, and 564, processing unit 110 may identify traffic lights that appear in the set of captured images and analyze the intersection geometry information. Based on the identification and analysis, processing unit 110 may cause one or more navigation responses in vehicle 200, as described above in connection with FIG. 5A .

[0124] FIG. 5E is a flowchart of an example process 500E for generating one or more navigation responses in vehicle 200 based on a vehicle path, according to a disclosed embodiment. In step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. The vehicle path may be represented using a set of points represented by coordinates (x, y), and the distance d between any two points in the set of points may be in the range of 1 to 5 meters. In one embodiment, processing unit 110 may construct the initial vehicle path using two polynomials, such as left and right road polynomials. Processing unit 110 may calculate the geometric midpoint between the two polynomials and offset each point included in the resulting vehicle path by a predetermined offset (e.g., a smart lane offset) if there is a predetermined offset (an offset of 0 may correspond to driving in the center of the lane). The offset may be in a direction perpendicular to the segment between any two points in the vehicle path. In another embodiment, processing unit 110 may use one polynomial and the estimated lane width to offset each point in the vehicle path by half the estimated lane width plus a predetermined offset (e.g., a smart lane offset).

[0125] At step 572, processing unit 110 may update the vehicle path constructed at step 570. Processing unit 110 may reconstruct the vehicle path constructed at 570 using a higher resolution such that the distance dk between two points in the set of points representing the vehicle path is shorter than the distance di described above. For example, the distance dk may be in the range of 0.1 to 0.3 meters. Processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm, which may result in a cumulative distance vector S corresponding to the total length of the vehicle path (i.e., based on the set of points representing the vehicle path).

[0126] In step 574, processing unit 110 calculates the look-ahead point ((x l ,z l ) in coordinates. The processing unit 110 may extract look-ahead points from the cumulative distance vector S, and the look-ahead points may be associated with a look-ahead distance and a look-ahead time. The look-ahead distance may have a lower bound range of 10 to 20 meters and may be calculated as the product of the speed of the vehicle 200 and the look-ahead time. For example, as the speed of the vehicle 200 decreases, the look-ahead distance may also decrease (e.g., until a lower bound is reached). The look-ahead time, which may range from 0.5 to 1.5 seconds, may be inversely proportional to the gain of one or more control loops associated with producing a navigation response in the vehicle 200, such as a heading error tracking control loop. For example, the gain of the heading error tracking control loop may depend on the bandwidth of the yaw rate loop, the steering actuator loop, and the vehicle lateral dynamics. Thus, the higher the gain of the heading error tracking control loop, the shorter the look-ahead time.

[0127] In step 576, processing unit 110 may determine a heading error and yaw rate command based on the look-ahead point identified in step 574. Processing unit 110 may calculate the arctangent of the look-ahead point, e.g., arctan(x l / z l) to identify the heading error. Processing unit 110 may determine the yaw rate command as the product of the heading error and a high-level control gain. The high-level control gain may be equal to (2 / look ahead time) if the look ahead distance is not at a lower limit. If the look ahead distance is at a lower limit, the high-level control gain may be equal to (2 x speed of vehicle 200 / look ahead distance).

[0128] 5F is a flowchart illustrating an example process 500F for determining whether a leading vehicle is changing lanes, according to the disclosed embodiments. In step 580, processing unit 110 may determine navigation information associated with the leading vehicle (e.g., a vehicle traveling in front of vehicle 200). For example, processing unit 110 may determine the position, velocity (e.g., direction and speed), and / or acceleration of the leading vehicle using the techniques described above in connection with FIGS. 5A and 5B. Processing unit 110 may also determine one or more road polynomials, lookahead points (associated with vehicle 200), and / or snail trails (e.g., a set of points describing the path taken by the leading vehicle) using the techniques described above in connection with FIG. 5E.

[0129] In step 582, processing unit 110 may analyze the navigation information identified in step 580. In one embodiment, processing unit 110 may calculate the distance (e.g., along the trail) between the snail trail and the road polynomial. If the difference in this distance along the trail exceeds a predetermined threshold (e.g., 0.1 to 0.2 meters for straight roads, 0.3 to 0.4 meters for gently curving roads, and 0.5 to 0.6 meters for sharply curving roads), processing unit 110 may determine that the leading vehicle is likely changing lanes. If multiple vehicles are detected traveling in front of vehicle 200, processing unit 110 may compare the snail trails associated with each vehicle. Based on the comparison, processing unit 110 may determine that a vehicle whose snail trail does not match the snail trails of the other vehicles is likely changing lanes. Processing unit 110 may further compare the curvature of the snail trail (associated with the leading vehicle) with the expected curvature of the road segment along which the leading vehicle is traveling. The expected curvature may be extracted from map data (e.g., data from map database 160), road polynomials, snail trails of other vehicles, prior knowledge about the road, etc. If the difference between the snail trail curvature and the expected curvature of the road segment exceeds a predetermined threshold, processing unit 110 may determine that the leading vehicle is likely changing lanes.

[0130] In another embodiment, processing unit 110 may compare the instantaneous position of the leading vehicle to a look-ahead point (associated with vehicle 200) over a specific time period (e.g., 0.5-1.5 seconds). If the difference in distance and the cumulative sum of discrepancies between the instantaneous position of the leading vehicle and the look-ahead point during the specific time period exceeds a predetermined threshold (e.g., 0.3-0.4 meters for straight roads, 0.7-0.8 meters for gently curving roads, and 1.3-1.7 meters for sharply curving roads), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the snail trail by comparing the lateral distance traveled along the trail to the expected curvature of the snail trail. The expected radius of curvature is calculated as: (δz 2 +δ x 2 ) / 2 / (δ x ) in which σ x represents the lateral movement distance, and σ z represents the longitudinal movement distance. If the difference between the lateral movement distance and the expected curvature exceeds a predetermined threshold (e.g., 500-700 meters), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the position of the leading vehicle. If the position of the leading vehicle obscures the road polynomial (e.g., the leading vehicle overlaps the road polynomial), processing unit 110 may determine that the leading vehicle is likely changing lanes. If the position of the leading vehicle is such that another vehicle is detected ahead of the leading vehicle and the snail trails of the two vehicles are not parallel, processing unit 110 may determine that the (closer) leading vehicle is likely changing lanes.

[0131] In step 584, processing unit 110 may determine whether leading vehicle 200 is changing lanes based on the analysis performed in step 582. For example, processing unit 110 may make that determination based on a weighted average of the individual analyses performed in step 582. Under such a scheme, for example, a determination by processing unit 110 that the leading vehicle is likely changing lanes based on a particular type of analysis may be assigned a value of “1” (with a “0” representing a determination that the leading vehicle is unlikely to be changing lanes). Different weights may be assigned to different analyses performed in step 582, and the disclosed embodiments are not limited to any particular combination of analyses and weights. Furthermore, in some embodiments, the analysis may utilize a trained system (e.g., a machine learning system or a deep learning system), which may, for example, estimate a subsequent path beyond the vehicle's current location based on images captured at the current location.

[0132] 6 is a flowchart illustrating an example process 600 for generating one or more navigational responses based on stereo image analysis, according to disclosed embodiments. In step 610, processing unit 110 may receive first and second pluralities of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122 and 124 having fields of view 202 and 204) may capture first and second pluralities of images of an area ahead of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first and second pluralities of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0133] At step 620, processing unit 110 may execute stereo image analysis module 404 to perform stereo image analysis of the first and second plurality of images to create a 3D map of the road in front of the vehicle and detect features in the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and road hazards. The stereo image analysis may be performed similarly to the steps described above in connection with FIGS. 5A-5D . For example, processing unit 110 may execute stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road marks, traffic lights, road hazards, etc.) in the first and second plurality of images, filter out a subset of the candidate objects based on various criteria, perform multi-frame analysis, construct measurements, and identify confidence levels for the remaining candidate objects. In performing the above steps, processing unit 110 may consider information from both the first and second plurality of images, rather than information from only one set of images. For example, processing unit 110 may analyze differences in pixel-level data (or other subsets of data from the two streams of captured images) for a candidate object that appears in both the first and second plurality of images. As another example, processing unit 110 may estimate the position and / or velocity (e.g., relative to vehicle 200) of a candidate object by observing that the object appears in one of the plurality of images but not in the other, or other differences that may exist for objects that appear in the two image streams. For example, the position, velocity, and / or acceleration for vehicle 200 may be determined based on the trajectory, location, movement characteristics, etc. of features associated with the object that appears in one or both of the image streams.

[0134] In step 630, processing unit 110 may execute navigation response module 408 to cause one or more navigational actions in vehicle 200 based on the analysis performed in step 620 and the techniques described above in connection with FIG. 4. The navigational responses may include, for example, turns, lane shifts, acceleration changes, speed changes, braking, etc. In some embodiments, processing unit 110 may use data derived from execution of speed and acceleration module 406 to cause one or more navigational responses. Furthermore, multiple navigational responses may be performed simultaneously, sequentially, or any combination thereof.

[0135] 7 is a flowchart illustrating an example process 700 for generating one or more navigational responses based on the analysis of three sets of images, according to disclosed embodiments. In step 710, processing unit 110 may receive first, second, and third pluralities of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture first, second, and third pluralities of images of an area in front of and / or to the side of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first, second, and third pluralities of images via three or more data interfaces. For example, each of image capture devices 122, 124, and 126 may have an associated data interface that communicates data to processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0136] At step 720, processing unit 110 may analyze the first, second, and third plurality of images to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and road hazards. The analysis may be performed similar to the steps described above in connection with FIGS. 5A-5D and 6. For example, processing unit 110 may perform monocular image analysis on each of the first, second, and third plurality of images (e.g., via execution of monocular image analysis module 402 and based on the steps described above in connection with FIGS. 5A-5D). Alternatively, processing unit 110 may perform stereo image analysis on the first and second plurality of images, the second and third plurality of images, and / or the first and third plurality of images (e.g., via execution of stereo image analysis module 404 and based on the steps described above in connection with FIG. 6). Processed information corresponding to the analysis of the first, second, and / or third plurality of images may be combined. In some embodiments, processing unit 110 may perform a combination of monocular image analysis and stereo image analysis. For example, processing unit 110 may perform monocular image analysis on the first plurality of images (e.g., via execution of monocular image analysis module 402) and stereo image analysis on the second and third plurality of images (e.g., via execution of stereo image analysis module 404). The configuration of image capture devices 122, 124, and 126—including their respective positions and fields of view 202, 204, and 206—may affect the type of analysis performed on the first, second, and third plurality of images. The disclosed embodiments are not limited to a particular configuration of image capture devices 122, 124, and 126 or the type of analysis performed on the first, second, and third plurality of images.

[0137] In some embodiments, processing unit 110 may perform tests on system 100 based on the images acquired and analyzed in steps 710 and 720. Such tests may provide an indicator of the overall performance of system 100 with a particular configuration of image capture devices 122, 124, and 126. For example, processing unit 110 may identify the rate of "false hits" (e.g., when system 100 incorrectly determines the presence of a vehicle or pedestrian) and "misses."

[0138] In step 730, processing unit 110 may cause one or more navigation responses in vehicle 200 based on information derived from two of the first, second, and third pluralities of images. The selection of two of the first, second, and third pluralities of images may depend on various factors, such as, for example, the number, type, and size of objects detected in each of the multiple images. Processing unit 110 may make the selection based on the quality and resolution of the images, the effective field of view reflected in the images, the number of captured frames, the degree to which one or more objects of interest actually appear in the frames (e.g., the percentage of frames in which the object appears, the proportion in which the object appears in each such frame, etc.), etc.

[0139] In some embodiments, processing unit 110 may select information derived from two of the first, second, and third plurality of images by identifying the extent to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 may combine processed information (whether monocular analysis, stereo analysis, or any combination of the two) derived from each of image capture devices 122, 124, and 126 to identify visual indicators (e.g., lane markings, detected vehicles and / or their positions and / or paths, detected traffic lights, etc.) that are consistent across images captured from each of image capture devices 122, 124, and 126. Processing unit 110 may also filter out information that is inconsistent across captured images (e.g., a vehicle changing lanes, a lane model showing a vehicle too close to vehicle 200, etc.). Thus, processing unit 110 may select information derived from two of the first, second, and third plurality of images based on the identification of consistent and inconsistent information.

[0140] The navigational responses may include, for example, turns, lane shifts, and acceleration changes. Processing unit 110 may generate one or more navigational responses based on the analysis performed in step 720 and the techniques described above in connection with FIG. 4 . Processing unit 110 may also generate one or more navigational responses using data derived from execution of velocity and acceleration module 406. In some embodiments, processing unit 110 may generate one or more navigational responses based on the relative position, relative velocity, and / or relative acceleration between vehicle 200 and an object detected in any of the first, second, and third plurality of images. The multiple navigational responses may be generated simultaneously, sequentially, or any combination thereof.

[0141] Analysis of the captured images and / or thermal maps may enable detection of specific characteristics of both parked and moving vehicles. Navigation changes may be calculated based on the detected characteristics. Embodiments for detecting specific characteristics based on one or more specific analyses of the captured images and / or thermal maps are described below with respect to Figures 8-28.

[0142] Car door opening event detection For example, identifying a vehicle and subsequently identifying a wheel component of the identified vehicle may enable targeted monitoring of door-opening events. Targeting the monitoring allows the system to identify and respond to door-opening events with a shorter reaction time than traditional motion detection. Embodiments of the present disclosure described below relate to systems and methods for detecting door-opening events using targeted monitoring.

[0143] 8 is an example functional block diagram of memory 140 and / or 150 that may be stored / programmed with instructions for performing one or more operations in accordance with the disclosed embodiments. While the following refers to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0144] 8 , memory 140 may store a vehicle side identification module 802, a wheel identification module 804, a door edge identification module 806, and a navigation response module 808. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 802-808 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Accordingly, any steps in the following processes may be performed by one or more processing devices.

[0145] In one embodiment, vehicle side identification module 802 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described below with respect to Figures 9-14, vehicle side identification module 802 may include instructions for determining a bounding box marking one or more vehicle sides.

[0146] In one embodiment, wheel identification module 804 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described below with respect to Figures 9-14, wheel identification module 804 may include instructions for determining ellipses marking one or more vehicle wheels.

[0147] In one embodiment, door edge identification module 806 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described below with respect to Figures 9-14, door edge identification module 806 may include instructions for identifying the appearance of a door edge and monitoring movement of the identified door edge.

[0148] In one embodiment, navigation response module 808 may store software executable by processing unit 110 to determine a desired navigation response based on data obtained from executing vehicle side identification module 802, wheel identification module 804, and / or door edge identification module 806. For example, navigation response module 808 may effect navigation changes according to method 1400 of FIG. 14, described below.

[0149] Additionally, any of the modules disclosed herein (e.g., modules 802, 804, and 806) may implement techniques related to trained systems (such as neural networks or deep neural networks) or untrained systems.

[0150] 9 is a schematic diagram of a road 902 from the perspective of a system included in a host vehicle (e.g., system 100 described above) according to disclosed embodiments. As shown in FIG. 9, road 902 may have one or more parked vehicles (e.g., parked vehicle 904 or parked vehicle 906).

[0151] System 100 can detect parked vehicles, for example, using an attention mechanism that returns suspicious patches and feeds the suspicious patches into a cascade of more complex classifiers to determine whether the patches are in fact vehicles. As described below, the attention mechanism and classifiers can be trained based on the true and false patches.

[0152] For example, system 100 can use an attention mechanism to detect the rear of a parked vehicle (e.g., rear of vehicle 908 and 910). Using the detected rear, system 100 can detect the sides of the parked vehicle (e.g., sides of vehicle 912 and 914). Once detected, the rear and / or sides of the vehicle can be tracked.

[0153] To detect the bounding box shown in FIG. 9 , the system 100 can input an image (e.g., an image from one of the image capture devices 122, 124, or 126) to one or more trained algorithms. The input image can be, for example, the original 1280x9560 grayscale image. The trained algorithm can output a scaled (e.g., 256x192) attention image. Suspect patches can be identified from the attention image, and the (x,y) points of the suspect patches can be scaled (e.g., by 5) and mapped onto the coordinates of the original image. The trained algorithm can use the (x,y) coordinates to determine the center of the suspect patch. The suspect patch can be scanned, for example, + / - 5 pixels in each direction, within the (x,y) coordinate range. In this example, each suspect patch produces a total of 11x11 candidate patches.

[0154] Further, in this example, each candidate patch is a square of size 2R+1, where R is the radius of the bounding ellipse. For example, as shown in Figure 10, a vehicle (e.g., vehicle 1002) may have an associated bounding ellipse (e.g., ellipse 1004) with a center (e.g., center 1006).

[0155] The candidate patches can be scaled to a regular size (such as 40x40) and used as input to one or more trained networks, such as one or more convolutional neural networks (CNNs) described below. For example, each of the one or more trained networks can score the input patches. Using the scores, a label can be assigned to each candidate patch based on the highest score. (In this example, the radius R and the original coordinates (x0, y0) can be used to map the candidate patches back to the original image.)

[0156] Each candidate patch with the highest score above a threshold (which may be preset or variable and learned from training) may be input to a final classifier. The final classifier may output the (x,y) coordinates of the bottom three points of the bounding box. These coordinates can be scaled back to the coordinates of the original image by multiplying by an appropriate factor. In the above example, a suitable scaling factor may be (2R+1) / 40. In addition to using the scaling factor, the actual position (x0,y0) can be added to the scaled coordinates (x,y). Using the unique labels, the system can identify which two of the three (x,y) coordinates belong to the sides (and to which sides), and which two of the three (x,y) coordinates belong to the back or front.

[0157] Those skilled in the art will appreciate that there may be variations to this example algorithm. For example, the size of the scaled attention image may vary, the shifting of the suspicious patch may vary, the size of the candidate patch may vary, etc. As a further example, the coordinate of the top of the bounding box may also be calculated. (In such an example, the final classifier may output the (x,y) coordinates of three additional points.) Additionally, other algorithms are possible in place of or in combination with the example algorithm above. For example, other algorithms may include different and / or additional classifiers.

[0158] As described above, the attention mechanism and subsequent classifier can be trained. For example, the training mechanism can utilize over one million example images, which can be, for example, 1280x960 grayscale images. In this example training set, the visible faces of the bounding box can be marked as left, right, back, or front. For example, the visible faces can be indicated as yellow, blue, red, and green, respectively. If a face is partially visible, only the obvious portion can be marked in the image and the partially invisible portion can be noted in the database.

[0159] In this training example, for each bounding box, the system can calculate the two furthest edges of the bounding box and construct a bounded ellipse centered midway between the two edges and with radius as the distance to the furthest edge.

[0160] In this training example, the system can then extract an attention image from the whole, which is a 256x192 (i.e., 5-fold) image. Each vehicle marked in the training image can be replaced by a point in the attention image located at the coordinates of the center of the ellipse divided by 5, and the value of that point can be the radius of the bounding ellipse.

[0161] In this training example, the example images can be used to train a convolutional neural network (CNN). Those skilled in the art will appreciate that other machine training techniques can be used instead of or in combination with the CNN. Thus, the neural network can map the original image to a sparse, reduced-resolution attention image. This approach can combine scene understanding (e.g., road location, image perspective) with local detection of any objects that look like cars. Other design choices are possible. For example, the network could first apply a filter bank designed to detect cars in places where cars are expected (e.g., not empty).

[0162] The neural network can send suspicious patches to a first classifier that can assign a score for each possible view. For example, the first classifier can assign one of four primary labels: left rear, left front, right rear, or right front. If only one face is visible, one of two possible labels can be randomly assigned. Each primary label can be further subdivided. For example, each primary label can be subdivided into whether the patch contains more "sides" than "ends," or vice versa. Such subdivision can be performed, for example, by comparing the image widths of the marked end and rear faces. If the widths are equal, the subdivision can be randomly assigned.

[0163] As a further example, each subdivision can be further divided into three subdivisions. For example, the "side" subdivision of the left rear label can include the following three subdivisions: "end" left rear 10 can refer to patches where the rear surface is 10% or less of the width of the left surface, "end" left rear 50 can refer to patches where the rear surface is more than 10% but less than 50%, and "end" left rear can refer to patches where the rear surface is more than 50%.

[0164] As a further example, if at least one face is hidden, each subdivision can be further labeled. In this example, there are 48 total combinations of subdivisions and labels. Those skilled in the art will appreciate that other divisions and labels resulting in the same or different total combinations are possible. For example, "side" rear left 20 can refer to patches where less than 20% of the face is on the left side; "side" rear left 50 can refer to patches where 20%-50% of the face is on the left side; "side" rear left 80 can refer to patches where 50%-80% of the face is on the left side; and "side" rear left 100 can refer to patches where 80%-100% of the face is on the left side.

[0165] As a further example, the four primary labels can be replaced by two primary labels, "side / end" and "end / side," depending on whether the side face appears to the left or right of the end face, respectively. The choice of division and labels depends on the amount of data available, as more subdivisions and labels require more examples for training.

[0166] Thus, the neural network can input patches centered on points in the attention map scaled back to the coordinates of the original image and shifted exhaustively by + / - 5 pixels in the x and y directions into the first classifier. Such shifting generates 121 shifted examples. Those skilled in the art will appreciate that other means of generating shifted examples are possible. For example, patches can be shifted exhaustively by + / - 4 (or + / - 3, etc.) pixels in the x and y directions.

[0167] In this example, each patch can be formed using the radius R of the bounding ellipse to cut a square of size (2R+1) x (2R+1) scaled to a regular size (e.g., 40x40 pixels). Those skilled in the art will appreciate that other means of generating patches are possible. For example, a square of size (2R-1) x (2R-1) can be cut. As a further example, one or more lengths of the bounding box can be used instead of 2R.

[0168] The neural network can input each labeled patch to a final classifier. The final classifier can output the (x,y) locations within the labeled patch of each of the three points that define the bottom of the bounding box. The output (x,y) coordinates can be relative to the patch. In some embodiments, a neural network can be trained for each combination of subdivision and label. In other embodiments, fewer neural networks can be trained.

[0169] Similar learning techniques can be used to train classifiers to extract more detailed features from identified vehicles. For example, classifiers can be trained to identify wheels, tires, "A" pillars, side mirrors, etc. In the example of Figure 11, wheels 1102 and 1104 of vehicle 1106 are marked with ellipses. The system can draw the ellipses, for example, by scaling the bounding box to a normal size (e.g., 40 pixels long by 20 pixels high) and inputting the scaled box into an appropriate classifier.

[0170] Based on identifying the wheels, the system can determine one or more "hot spots" on the identified vehicle where a door-opening event may be expected to occur. For example, the one or more hot spots may be located between the identified tires and / or on the identified rear tires. As shown in FIG. 12A , the one or more hot spots may be monitored for the appearance of vertical stripes 1202 on the vehicle 1204. The appearance of the stripes 1202 may indicate the start of a door-opening event.

[0171] In some embodiments, the system may use one or more features on the vehicle 1204 as reference points to track the movement of the edge of the stripe 1202 relative to the side of the vehicle 1204. For example, the one or more features may include an identified tire or other identified feature such as the front end of the vehicle 1204, one or more tail lights of the vehicle 1204, etc.

[0172] 12B, when a door of vehicle 1204 opens, stripe 1202 may unfold. Edge 1202a of stripe 1202 may be fixed in position along the body of vehicle 1204, while edge 1202b of stripe 1202 may appear to move in front of vehicle 1204. Thus, the system can determine that there is a door-opening event based on monitoring stripe 1202.

[0173] The host vehicle can be subjected to navigation changes based on the presence of a door opening event. For example, as shown in Figure 13, the yaw 1301 of the host vehicle has changed, indicating that the host vehicle is moving away from the door opening event.

[0174] 14 is a flowchart illustrating an example process 1400 for generating one or more navigational responses based on detecting a door-opening event, according to the disclosed embodiments. In step 1402, processing unit 110 may receive at least one image of the host vehicle's environment via data interface 128. For example, a camera included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture at least one image of an area in front of and / or to the side of the host vehicle and transmit them to processing unit 110 over a digital connection (e.g., USB, wireless, Bluetooth, etc.).

[0175] In step 1404, processing unit 110 may analyze the at least one image to identify a side of the parked vehicle. Step 1404 may further include associating at least one bounding box with the shape of the side of the parked vehicle. For example, the analysis may be performed using the learned algorithm discussed above with respect to FIGS. 9 and 10.

[0176] In step 1406, processing unit 110 may identify structural features of the parked vehicle. In some embodiments, processing unit 110 may identify a first structural feature of the parked vehicle in a forward region of the side of the parked vehicle and a second structural feature of the parked vehicle in a rear region of the side of the parked vehicle. For example, the structural features may include wheel components (tires, hubcaps, wheel structures, etc.), mirrors, "A" pillars, "B" pillars, "C" pillars, etc. The first structural feature and / or the second structural feature may be identified in a region near the identified side. For example, the analysis may be performed using the learned algorithm discussed above with respect to FIG. 11 .

[0177] In step 1408, processing unit 110 may identify a door edge of the parked vehicle. The door edge may be identified within a region near a structural feature. For example, in an embodiment in which processing unit 110 identifies front and rear wheel components, the vicinity of the first and second wheels may include a region between the front and rear wheel components. As a further example, in an embodiment in which processing unit 110 identifies front and rear wheel components, the vicinity of the first and second wheel components may include a region above the rear wheel components. The analysis may be performed using the learned algorithm discussed above with respect to FIG. 12A.

[0178] In step 1410, processing unit 110 may determine a change in image characteristics of the door edge. For example, processing unit 110 may monitor at least two images received from the image capture device for the appearance of a vertical stripe near the first wheel component and the second wheel component, as discussed above with respect to FIG. 12B . In this example, a first edge of the vertical stripe is fixed along the body of the parked vehicle in the monitored image, and a second edge of the vertical stripe moves toward a front region of the parked vehicle in the monitored image. After appearance, the width of the door edge (i.e., the width of the vertical stripe) may be tracked over time. In some embodiments, the change in image characteristics of the door edge may include a widening of the door edge. In such embodiments, determining the change in image characteristics of the door edge may include monitoring the widening of the vertical stripe.

[0179] In some embodiments, processing unit 110 can estimate the amount the door will open. For example, processing unit 110 can extend a column of fixed edges to intersect with the bounding box and use the y-coordinate of the intersection to estimate the distance to the door. In this example, the distance to the door and the width of the stripe can be used to estimate the amount the door will open. Thus, processing unit 110 can determine the distance the door edge extends away from the parked vehicle based on the determined width. Thus, the separation of the door edge from the body of the vehicle (i.e., the amount the door opens) can be tracked over time.

[0180] At step 1412, processing unit 110 may alter the navigational route of the host vehicle. For example, the navigational response may include a turn, a lane shift, a change in acceleration, etc. (as shown in FIG. 13 ). Processing unit 110 may generate one or more navigational responses based on the calculations made at step 1410. For example, processing unit 110 may move the host vehicle away from the door edge event and / or slow the host vehicle in response to the door edge event. In this example, processing unit 110 may determine a lateral safety distance for the host vehicle based on a determined distance that the door edge extends away from the parked vehicle, and the alteration of the host vehicle's navigational route may be based at least in part on the determined lateral safety distance. In another example, processing unit 110 may determine the lateral safety distance for the host vehicle based on a predetermined value, such as a value consistent with a typical degree of protrusion associated with a vehicle door opening. Further, by way of example, different default values ​​may be used for different types of vehicles, e.g., a longer default safety distance value may be used for vehicles of other sizes, such as trucks, compared to a safety distance value used for small vehicles.

[0181] Processing unit 110 may also generate one or more navigational responses using data obtained from executing velocity and acceleration module 406. Multiple navigational responses may occur simultaneously, sequentially, or any combination thereof. For example, the navigational responses may be determined by a trained system. Further, by way of example, the trained system may be configured to avoid violating certain safety constraints while optimizing performance, and may be configured to react to detecting a door opening by invoking a navigation change. In another example, a set of rules may be used to determine a desired response upon detecting a door opening event (of a parked vehicle).

[0182] Detecting vehicles entering the host vehicle's lane Systems and methods for identifying road homographies and identifying vehicle wheel components may enable targeted surveillance of vehicle movement. Targeting surveillance enables the system to identify and respond to movement into the host vehicle's lane from another lane or from a parking position with shorter reaction times than traditional motion detection, at least under certain circumstances. Embodiments of the present disclosure described below relate to systems and methods for detecting vehicles entering the host vehicle's lane using targeted surveillance.

[0183] 15 is an exemplary functional block diagram of memory 140 and / or 150 that may be stored / programmed with instructions for performing one or more operations in accordance with the disclosed embodiments. While the following is directed to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0184] 15, memory 140 may store road homography module 1502, wheel identification module 1504, motion analysis module 1506, and navigation response module 1508. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 1502-1508 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Accordingly, any steps in the following processes may be performed by one or more processing devices.

[0185] In one embodiment, road homography module 1502 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, warp the homography of a road in one or more images acquired by one of image capture devices 122, 124, and 126. For example, road homography module 1502 may include instructions for performing method 1800 of FIG. 18, described below.

[0186] In one embodiment, wheel identification module 1504 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described above with respect to Figures 9-14, wheel identification module 1504 may include instructions for determining an oval to mark one or more vehicle wheels.

[0187] In one embodiment, motion analysis module 1506 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, perform analysis of one or more images acquired by one of image capture devices 122, 124, and 126 (and / or one or more images processed by road homography module 1502) to track the movement of one or more identified vehicle components. For example, in combination with wheel identification module 1504, motion analysis module 1506 may track the movement of identified wheel components of a vehicle over time.

[0188] In one embodiment, navigation response module 1508 may store software executable by processing unit 110 to determine a desired navigation response based on data obtained from executing road homography module 1502, wheel identification module 1504, and / or motion analysis module 1506. For example, navigation response module 1508 may effect navigation changes according to method 1700 of FIG. 17, described below.

[0189] Additionally, any of the modules disclosed herein (e.g., modules 1502, 1504, and 1506) may implement techniques related to trained systems (such as neural networks or deep neural networks) or untrained systems.

[0190] Figure 16A shows a parked car 1602 from the perspective of a system according to the disclosed embodiments. In the example of Figure 16A, wheels 1604 and 1606 of the vehicle 1602 are marked with ellipses. The system can draw the ellipses, for example, by scaling the bounding box to a normal size (e.g., 40 pixels long by 20 pixels high) as described above and inputting the scaled box into an appropriate classifier.

[0191] Figure 16B shows vehicle 1602 leaving a parked (i.e., stationary) state. In the example of Figure 16B, wheels 1604 and 1606 of vehicle 1602 are rotating as vehicle 1602 moves. The system can track the rotation of wheels 1604 and 1606, as discussed below, to determine that vehicle 1602 is moving.

[0192] 17 is a flowchart illustrating an example process 1700 for generating one or more navigational responses based on detecting a target vehicle entering the lane of a host vehicle, according to a disclosed embodiment. In step 1702, processing unit 110 may receive multiple images of the host vehicle's environment via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture multiple images of the area in front of and / or to the sides of the host vehicle and transmit them to processing unit 110 over a digital connection (e.g., USB, wireless, Bluetooth, etc.).

[0193] The plurality of images can be collected over time. For example, the plurality can include a first image captured at time t=0, a second image captured at t=0.5 seconds, and a third image captured at t=1.0 seconds. The timing between images can depend at least on the scan rate of the one or more image capture devices.

[0194] In step 1704, processing unit 110 may analyze the at least one image to identify a target vehicle. For example, identifying the target vehicle may further include associating at least one bounding box with a shape of a side of the target vehicle. For example, the analysis may be performed using the learned algorithm discussed above with respect to FIGS. 9 and 10. In some embodiments, step 1704 may further include identifying a wheel on the side of the identified target vehicle. For example, the analysis may be performed using the learned algorithm discussed above with respect to FIG. 11.

[0195] Step 1704 is not limited to wheels and may also include wheel components. For example, processing unit 110 may identify wheel components including at least one of a tire, a hubcap, or a wheel structure.

[0196] In step 1706, processing unit 110 may identify movement associated with the identified wheel. For example, the processing unit may identify movement within a region including at least one wheel component of the target vehicle, which may include a region adjacent to the road surface. By monitoring at least two of the multiple images, the processing unit may identify movement using an indication of rotation of the at least one wheel component.

[0197] As a further example, processing unit 110 may identify at least one feature associated with at least one wheel component (e.g., a logo on the wheel, a measurement of the tire of the wheel, a measurement of the hubcap of the wheel, a particular patch of pixels). Using the at least one feature, processing unit 110 may identify an indicator of a position change of the at least one feature (e.g., blurring of the logo, changed coordinates of the patch of pixels).

[0198] As a further example, processing unit 110 may perform a homography transformation of the road as described below with respect to method 1800 of FIG. 18, identify a point of contact (which may be stationary) between the identified wheel and the transformed road, and track points on that point of contact to identify movement.

[0199] The processing unit may determine a speed of movement of the target vehicle using the indicators of the rotation, the position change of the at least one feature, and / or the tracked points. Processing unit 110 may use the determined speed in making navigation changes in step 1708, described below. Additionally, processing unit 110 may estimate a distance to the tires using a ground plane constraint and may estimate lateral movement of the target vehicle based on the estimated distance.

[0200] At step 1708, processing unit 110 may cause a navigational change for the host vehicle. For example, the navigational response may include a change in the host vehicle's direction of travel (as shown in FIG. 13 above), a lane shift, a change in acceleration (e.g., applying the brakes of the host vehicle), etc. Processing unit 110 may cause one or more navigational responses based on the determination made at step 1706. For example, processing unit 110 may move the host vehicle away from the target vehicle and / or slow down the host vehicle in response to the movement of the target vehicle. In this example, processing unit 110 may determine a lateral safety distance for the host vehicle based on the calculated speed (and / or estimated lateral movement) of the target vehicle, and a change in the host vehicle's navigational path may be based at least in part on the determined lateral safety distance.

[0201] Processing unit 110 may also generate one or more navigational responses using data obtained from executing velocity and acceleration module 406. The multiple navigational responses may occur simultaneously, sequentially, or any combination thereof.

[0202] 18 is a flowchart illustrating an example process 1800 for performing a homography transformation of a road. In step 1802, processing unit 110 may receive multiple images of the host vehicle's environment via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture multiple images of the area in front of and / or to the sides of the host vehicle and transmit them over a digital connection (e.g., USB, wireless, Bluetooth, etc.) to processing unit 110.

[0203] The multiple images can be collected over time. For example, the multiple can include a first image captured at time t=0 and a second image captured at t=0.5 seconds. The timing between images can depend at least on the scan rate of the one or more image capture devices.

[0204] In step 1804, processing unit 110 may first transform a first image of the plurality of images relative to a second image of the plurality of images. For example, one of the first image or the second image may be rotated based on an estimate of the yaw, pitch, and roll of the host vehicle.

[0205] In step 1806, processing unit 110 may select a grid of points in the first image or the second image as a grid of reference points. For example, this grid may be formed from any shape, such as an ellipse, a rectangle, a trapezoid, etc. Alternatively, a random distribution of points may be selected.

[0206] In step 1808, processing unit 110 may arrange patches around the selected grid. For example, the patches may be of uniform size and shape. Alternatively, the patches may be non-uniform and / or randomly non-uniform.

[0207] In step 1810, processing unit 110 may track points of the selected grid, for example, using a patch-based normalized correlation calculation. From the tracked points, processing unit 110 may select a subset of points with the highest scores based on the tracking.

[0208] At step 1812, processing unit 110 may fit the tracked points to a homography. In some embodiments, processing unit 110 may compute multiple homographies using random subsets of the tracked points. In such embodiments, processing unit 110 may retain the random subset with the highest scoring homography.

[0209] In step 1814, processing unit 110 may correct the initial transformation using the homography from step 1812. For example, processing unit 110 may use a random subset of points with the highest scoring homography to retransform a first image of the plurality of images to a second image of the plurality of images. Processing unit 110 may further directly compute a least squares homography from the retransformed images. Those skilled in the art will understand that other algorithms for computing road homographies may be used.

[0210] One-way road detection based on the direction of parked vehicles Systems and methods for identifying vehicles and identifying the front and / or rear of the identified vehicle may enable the detection of one-way roads. By using the identified front and / or rear of the vehicle, one-way roads can be detected without the need to interpret signs or even when no vehicles are moving on the road. Embodiments of the present disclosure described below relate to systems and methods for detecting one-way roads based on the direction of parked vehicles.

[0211] 19 is an exemplary functional block diagram of memory 140 and / or 150 that may be stored / programmed with instructions for performing one or more operations in accordance with the disclosed embodiments. While the following is directed to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0212] 19 , memory 140 may store a vehicle identification module 1902, a direction determination module 1904, a vehicle side identification module 1906, and a navigation response module 1908. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 1902-1908 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Accordingly, any steps of the following processes may be performed by one or more processing devices.

[0213] In one embodiment, vehicle identification module 1902 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described above with respect to Figures 9-14, vehicle identification module 1902 may include instructions for determining bounding boxes for one or more vehicles.

[0214] In one embodiment, direction determination module 1904 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described below with respect to Figures 20A and 20B, direction determination module 1904 may include instructions for determining the forward direction of an identified vehicle.

[0215] For example, the forward direction may indicate whether an identified parallel parked vehicle is facing toward the host vehicle or away from it. As a further example, the forward direction or "inclined direction" may indicate whether an identified vehicle parked in a diagonal spot is leaning toward the host vehicle or away from it. In such an example, the forward direction may further indicate whether the identified vehicle is backing up to the diagonal spot or moving forward into it.

[0216] In one embodiment, vehicle side identification module 1906 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described above with respect to Figures 9-14, vehicle side identification module 1906 may include instructions for classifying identified bounding boxes of one or more vehicles.

[0217] In one embodiment, navigation response module 1908 may store software executable by processing unit 110 to determine a desired navigation response based on data obtained from executing vehicle identification module 1902, direction determination module 1904, and / or vehicle side identification module 1906. For example, navigation response module 1908 may effect navigation changes according to method 2100 of FIG. 21 described below.

[0218] Additionally, any of the modules disclosed herein (e.g., modules 1902, 1904, and 1906) may implement techniques related to trained systems (such as neural networks or deep neural networks) or untrained systems.

[0219] 20A shows a one-way road 2002 from the perspective of a system according to disclosed embodiments. The road 2002 can include a first plurality of stopped vehicles (e.g., first vehicle 2004) on one side and a second plurality of stopped vehicles (e.g., second vehicle 2006) on the other side. As described below with respect to method 2100 of FIG. 21 , the system can determine the forward direction of the first plurality of vehicles and the forward direction of the second plurality of vehicles.

[0220] 20A , the system may determine that road 2002 is a one-way road if the forward direction of both the first and second plurality of vehicles is the same. In other embodiments where one side of road 2002 is parked diagonally rather than parallel, the system may determine that road 2002 is a one-way road if the forward direction of the parallel parked side is the same as the slope direction of the diagonally parked side. In still other embodiments where road 2002 has two diagonally parked sides, the system may determine that road 2002 is a one-way road if the slope direction of both the first and second plurality of vehicles is the same. In certain aspects, this determination may depend on whether the first and / or second plurality of vehicles are parked back into or forward into a diagonal spot.

[0221] FIG. 20B also illustrates a one-way road 2008 from the perspective of a system according to disclosed embodiments. Road 2008, like road 2002, can include a first plurality of stopped vehicles (e.g., first vehicle 2010) on one side and a second plurality of stopped vehicles (e.g., second vehicle 2012) on the other side. As described below with respect to method 2100 of FIG. 21, the system can determine the forward direction of the first plurality of vehicles and the forward direction of the second plurality of vehicles. As shown in FIG. 20A, the system can determine that a vehicle (e.g., vehicle 2010) is improperly parked if the forward direction of the vehicle differs from the forward direction of a plurality of vehicles associated with the vehicle. For example, the system can determine that a vehicle is improperly parked if the number of other vehicles in the associated plurality that have different forward directions exceeds a threshold. As a further example, the system may determine that a vehicle is correctly parked if the ratio of other vehicles with different forward directions to vehicles with the same forward direction is above a threshold (e.g., 50% or more, 60% or more, 70% or more, etc.). In some embodiments, this determination may be used to issue a traffic ticket to the owner or operator of vehicle 2010 (or to instruct the operator of the host vehicle to issue a traffic ticket to the owner or operator of vehicle 2010).

[0222] 21 is a flowchart illustrating an example process 2100 for generating one or more navigation responses based on detecting whether a road on which a host vehicle is traveling is a one-way road, according to a disclosed embodiment. In step 2102, processing unit 110 may receive at least one image of the host vehicle's environment via data interface 128. For example, a camera included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture at least one image of an area in front of and / or to the side of the host vehicle and transmit them to processing unit 110 over a digital connection (e.g., USB, wireless, Bluetooth, etc.).

[0223] In step 2104, processing unit 110 may analyze at least one image to identify a first plurality of vehicles on one side of the road. For example, identifying the first plurality of vehicles may further include associating bounding boxes with shapes of the first plurality of vehicles. For example, the analysis may be performed using the learned algorithm discussed above with respect to FIGS. 9 and 10.

[0224] Step 2104 may further include identifying a side of at least one of the first plurality of vehicles or at least one of the second plurality of vehicles based on an analysis of the at least one image. For example, the analysis may be performed using the learned algorithm discussed above with respect to FIGS. 9 and 10. In some embodiments, identifying the side may be based on at least two features associated with at least one of the first plurality of vehicles or at least one of the second plurality of vehicles. For example, the features associated with the vehicles may include mirrors, windows, door handles, door shape, number of doors, slope of the windshield and / or rear window, etc. In some embodiments, the identified side may be a right side. In other embodiments, the identified side may be a left side.

[0225] In step 2106, the processing unit may analyze the at least one image to identify a second plurality of vehicles on the other side of the road. Step 2106 may be performed similarly to and / or simultaneously with step 2104.

[0226] In step 2108, processing unit 110 may determine a first forward direction for the first plurality of vehicles. In some embodiments, the first plurality of vehicles may all have the same forward direction. In other embodiments, the forward directions may be different.

[0227] In step 2110, processing unit 110 may determine a second forward direction for a second plurality of vehicles. In some embodiments, the second plurality of vehicles may all have the same forward direction. In other embodiments, the forward directions may be different.

[0228] In step 2112, processing unit 110 may cause a navigational change for the host vehicle. For example, the navigational response may include a turn (as shown in FIG. 13 above), a lane shift, a change in acceleration (e.g., braking the host vehicle), etc. Processing unit 110 may cause one or more navigational responses based on the calculations made in step 2112. For example, processing unit 110 may determine that the road is a one-way road based on the first forward direction and the second forward direction. Based on this determination, processing unit 110 may slow down or stop the host vehicle and / or make a U-turn.

[0229] In some embodiments, method 2100 may include additional steps. For example, method 2100 may include receiving navigation instructions for navigating the host vehicle from a first road on which the host vehicle is traveling to a second road. The navigation instructions may include instructions to turn the host vehicle onto the second road, instructions to turn onto the second road, instructions to merge onto a ramp onto the second road, etc.

[0230] In such an embodiment, method 2100 may further include determining that both the first forward direction and the second forward direction are opposite to a direction of travel in which the host vehicle will travel if the host vehicle turns onto the second road. For example, processing unit 110 may analyze an image of the vehicle on the second road to determine the first forward direction and the second forward direction, and then determine whether the forward directions are opposite to the predicted direction of travel of the host vehicle. In response to determining that both the first forward direction and the second forward direction are opposite to the direction of travel in which the host vehicle will travel if the host vehicle turns onto the second road, processing unit 110 may abort the navigation instructions. For example, processing unit 110 may abort turning onto the second road, merging onto the second road, etc. because it has determined that the road is a one-way road opposite to the predicted direction of travel.

[0231] In further embodiments, method 2100 may include receiving an override command to restore the interrupted navigation instructions. For example, the override command may be initiated based on manual confirmation received from a person in the host vehicle, based on accessing map data, based on crowd-sourced data related to the driving direction of the second road, etc.

[0232] Processing unit 110 may also generate one or more navigational responses using data obtained from executing velocity and acceleration module 406. The multiple navigational responses may occur simultaneously, sequentially, or any combination thereof.

[0233] Predicting the state of parked vehicles based on thermal profiles Systems and methods for predicting the state of a parked vehicle based on thermal profiles may enable prediction of vehicle movement before the parked vehicle begins to move. In this way, rather than waiting to actually detect movement as in traditional motion detection, the system can identify predicted movement and preemptively adjust accordingly. The embodiments of the present disclosure described below relate to systems and methods for predicting the state of a parked vehicle based on thermal profiles.

[0234] 22 is an exemplary functional block diagram of memory 140 and / or 150 that may be stored / programmed with instructions for performing one or more operations in accordance with the disclosed embodiments. While the following is directed to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0235] 22 , memory 140 may store a visual-infrared alignment module 2202, a vehicle identification module 2204, a state prediction module 2206, and a navigation response module 2208. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 2202-2208 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Accordingly, any steps in the following processes may be performed by one or more processing devices.

[0236] In some embodiments, visual-to-infrared alignment module 2202 may store instructions (such as computer vision software) that, when executed by processing unit 110, align one or more visible light images acquired by one of image capture devices 122, 124, and 126 with one or more infrared images (i.e., thermal maps) acquired by one of image capture devices 122, 124, and 126. For example, visual-to-infrared alignment module 2202 may include instructions for performing method 2500 of FIG. 25, described below.

[0237] In other embodiments, vehicle identification module 2204 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described below with respect to Figures 9-14, vehicle identification module 2202 may include instructions for determining bounding boxes for one or more vehicles.

[0238] In one embodiment, state prediction module 2206 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, perform analysis of one or more aligned images from visual-infrared alignment module 2202 to predict a state of one or more identified vehicles. For example, state prediction module 2206 may output a predicted state based on visual and thermal indicators of the identified vehicles.

[0239] In one embodiment, navigation response module 2208 may store software executable by processing unit 110 to determine a desired navigation response based on data obtained from executing visual-infrared alignment module 2202, vehicle identification module 2204, and / or condition prediction module 2206. For example, navigation response module 2208 may effect navigation changes according to method 2400 of FIG. 24, described below.

[0240] Additionally, any of the modules disclosed herein (e.g., modules 2202, 2204, and 2206) may implement techniques related to trained systems (such as neural networks or deep neural networks) or untrained systems.

[0241] 23A illustrates a parked vehicle 2302 from the perspective of a system according to disclosed embodiments. For example, the vehicle 2302 may be monitored by the system for changing lighting conditions and / or temperature characteristics. The system may determine a predicted state of the vehicle 2302 based on the changing lighting conditions and / or temperature characteristics, for example, using method 2400 of FIG. 24.

[0242] Figure 23B shows a parked vehicle 2302 having a change in lighting state. In the example of Figure 23C, tail lights 2304a and 2304b of vehicle 2302 change from an unlit state to an lit state. Other embodiments are possible in which the headlights indicate a change in lighting state. Additionally, other embodiments are possible in which the vehicle's headlights and / or tail lights change from an lit state to an unlit state.

[0243] FIG. 23C shows (from different angles) a parked vehicle 2302 with a warm engine 2306 and cold tires 2308a and 2308b. As used herein, “warm” and “cold” refer to deviations from expected temperature values, which may be predetermined and / or learned. For example, an engine may be “warm” if it is above ambient temperature and “cold” if it is at or below ambient temperature. In embodiments of the present disclosure, references to the engine or engine temperature may relate to a particular area of ​​the vehicle whose temperature is typically affected by the engine temperature, such as the hood located at the front of the vehicle. “Warm” and “cold” temperature thresholds may be selected to reflect the expected temperature of the hood under particular conditions, possibly with some margin to reduce false positive or false negative detections as needed. In one example, the threshold temperature may be adjusted to account for heating from the sun under clear conditions (e.g., after determining that the vehicle's hood is exposed to the sun). In another example, the effect of the sun may be incorporated into the vehicle's color, which may be determined by spectral analysis of the image. In another example, the threshold value may be determined, for example, by averaging previously detected temperatures of parked cars and / or temperatures of specific areas of one or more parked cars, possibly within a local area of ​​the car currently being monitored.

[0244] Similarly, tire temperature can be used as an indication of the vehicle's condition. For example, a tire can be "warm" if it is above road temperature and "cold" if it is at or below road temperature. In another example, the threshold temperature (e.g., the temperature used to distinguish between "warm" and "cold" tires) can be related to an expected or calculated tire operating temperature. The operating temperature calculation can take into account ambient conditions and possibly a model of driving and its effect on tire temperature. In yet another example, the operating temperature calculation can also include a tire cooling model. The tire cooling model can also take ambient conditions into account. In the example of FIG. 23C, the system can determine a predicted state of the vehicle 2302 based on the temperatures of the engine 2306 and tires 2308a and 2308b, for example, using method 2400 of FIG. 24.

[0245] 24 is a flowchart illustrating an example process 2400 for determining a predicted state of a parked vehicle according to the disclosed embodiments. In step 2402, processing unit 110 may receive multiple images of the host vehicle's environment via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture multiple images of the area in front of and / or to the sides of the host vehicle and transmit them over a digital connection (e.g., USB, wireless, Bluetooth, etc.) to processing unit 110.

[0246] The plurality of images can be collected over time. For example, the plurality can include a first image captured at time t=0, a second image captured at t=0.5 seconds, and a third image captured at t=1.0 seconds. The timing between images can depend at least on the scan rate of the one or more image capture devices.

[0247] In step 2404, processing unit 110 may analyze the plurality of images to identify parked vehicles. For example, identifying the parked vehicles may further include associating at least one bounding box with a shape of a side of the parked vehicle. For example, the analysis may be performed using the learned algorithm discussed above with respect to FIGS. 9 and 10. In some embodiments, step 2404 may further include identifying an engine at the front of the identified target vehicle and / or wheels on the side of the identified target vehicle. For example, the analysis may be performed using the learned algorithm discussed above with respect to FIG. 11.

[0248] In step 2406, processing unit 110 may analyze the multiple images to identify changes in lighting conditions for the parked vehicle. For example, processing unit 110 may identify tail lights and / or head lights for the parked vehicle and monitor the identified tail lights and / or head lights for changes from off to on or from on to off (as seen in the example of FIG. 23B). Other embodiments may include more detailed changes, such as a change from only parking lights on to brake lights on.

[0249] In some embodiments, method 2400 may include determining a predicted state of the parked vehicle based on the change in lighting conditions and may proceed directly to step 2412 (i.e., causing at least one navigation response by the host vehicle based on the predicted state of the parked vehicle). For example, processing unit 110 may determine that the predicted state of the parked vehicle includes an indication that the parked vehicle's engine has been started based on the lighting state of at least one light associated with the parked vehicle changing from an unlit state to an on state. Similarly, processing unit 110 may determine that the predicted state of the parked vehicle includes an indication that the parked vehicle's engine has been turned off based on the lighting state of at least one light associated with the parked vehicle changing from an on state to an on state. Thus, in such embodiments, these determinations may not include using thermal imagery.

[0250] In step 2408, processing unit 110 may receive at least one thermal image (i.e., infrared image) of the host vehicle's environment via data interface 128. For example, a camera included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture at least one thermal image of an area in front of and / or to the side of the host vehicle and transmit them to processing unit 110 over a digital connection (e.g., USB, wireless, Bluetooth, etc.).

[0251] Step 2408 can be performed separately or simultaneously with step 2402. Thus, in some embodiments, the visible light image and the infrared image may be received simultaneously. In other embodiments, different scan rates and / or different transmission speeds between the one or more image capture devices and the one or more infrared image capture devices may result in a delay between the visible light image and the infrared image.

[0252] In some embodiments, method 2400 may further include registering at least one of the plurality of images with the at least one thermal image. Based on the registering, method 2400 may further include identifying at least one of an engine region or at least one wheel component region of the parked vehicle within the registered at least one thermal image.

[0253] In step 2410, processing unit 110 may determine a predicted state of the parked vehicle based on the analysis of step 2406 and / or an analysis of the at least one thermal image. For example, the predicted state may include an indication that the parked vehicle's engine has been started or an indication that the parked vehicle's engine has been turned off. As a further example, the predicted state may include an indication that the parked vehicle is not expected to move within a predetermined period of time, an indication that the parked vehicle is expected to move within a predetermined period of time, or an indication that the parked vehicle's door is expected to open within a predetermined period of time.

[0254] In some embodiments, analyzing the at least one thermal image may include determining a temperature value of an engine area of ​​the parked vehicle. For example, if the temperature value is below a threshold, processing unit 110 may determine a predicted state including an indication that the parked vehicle is not expected to move within a predetermined period of time. The predetermined period of time may depend, for example, on known characteristics of the parked vehicle or the parked vehicle's engine.

[0255] In some embodiments, analyzing the at least one thermal image may include determining a first temperature value of an engine area of ​​the parked vehicle and a second temperature value of at least one wheel component of the parked vehicle. In such embodiments, a predicted state of the parked vehicle may be determined based on comparing the first temperature value to a first threshold and comparing the second temperature value to a second threshold. For example, if the first temperature value is above the first threshold and the second temperature value is below the second threshold, processing unit 110 may determine a predicted state that includes an indication that the parked vehicle is expected to move within a predetermined time period. As a further example, if the first temperature value is above the first threshold and the second temperature value is above the second threshold, processing unit 110 may determine a predicted state that includes an indication that a door of the parked vehicle is expected to open within a predetermined time period.

[0256] In some embodiments, processing unit 110 may perform additional monitoring of the received images based at least in part on the predicted state, depending on whether the predicted state is based on changes in illumination, analysis of at least one thermal image, or a combination thereof. For example, if the predicted state indicates that the parked vehicle's engine has been turned off and / or that the parked vehicle's door is expected to open within a predetermined time period, processing unit 110 may monitor one or more portions of the received images for changes in image characteristics of the parked vehicle's door edge. An example of this monitoring is described above with respect to method 1400 of FIG. 14. As a further example, if the predicted state indicates that the parked vehicle's engine has been started and / or that the parked vehicle is expected to move within a predetermined time period, processing unit 110 may monitor one or more wheel components of the received images for movement of the wheel components. An example of this monitoring is described above with respect to method 1700 of FIG. 17.

[0257] At step 2412, processing unit 110 may cause a navigational change for the host vehicle. For example, a navigational response may include a change in direction of travel of the host vehicle, a lane shift, a change in acceleration (as shown in FIG. 13 above), etc. Processing unit 110 may cause one or more navigational responses based on the predicted state determination made at step 2410. For example, processing unit 110 may move the host vehicle away from the target vehicle and / or slow down the host vehicle in response to a predicted state indicating that the parked vehicle is expected to move within a predetermined time period or a predicted state indicating that a door of the parked vehicle is expected to open within a predetermined time period.

[0258] 25 is a flowchart showing an example process 2500 for aligning a visible light image and an infrared image. In steps 2502 and 2504, processing unit 110 may receive at least one visible light image of the host vehicle's environment and at least one infrared image of that environment via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture at least one visible light image and at least one infrared image of an area in front of and / or to the side of the host vehicle and transmit them to processing unit 110 over a digital connection (e.g., USB, wireless, Bluetooth, etc.).

[0259] In some embodiments, the visible light image and the infrared image may be received simultaneously, while in other embodiments, different scan rates and / or different transmission speeds between the one or more image capture devices and the one or more infrared image capture devices may result in a delay between the visible light image and the infrared image.

[0260] In step 2506, processing unit 110 may select a set of reference points in the at least one infrared image. For example, the set of reference points may be selected randomly or may include identification of known objects (e.g., pedestrians, trees, vehicles, etc.) based on known characteristics.

[0261] In step 2508, processing unit 110 may project the reference points from the at least one infrared image onto the at least one visible light image. For example, processing unit 110 may project a shape (e.g., an ellipse, a rectangle, a trapezoid, etc.) that represents (e.g., surrounds) the reference points onto locations in the at least one visible light image.

[0262] In step 2510, processing unit 110 may optimize gain and / or exposure for portions of the visible light image corresponding to the reference points. For example, improving contrast through optimization may result in more reliable alignment. Step 2508 is optional and need not be performed in all embodiments.

[0263] In step 2512, processing unit 110 may align at least one infrared image with at least one visible light image. For example, aligning (or matching) the images may include searching along epipolar lines over a distance that optimizes an alignment measure. In this example, optimizing the alignment measure may ensure that the distance between a reference point and the viewer and / or the distance between a reference point and other objects is the same in both the visible light image and the infrared image.

[0264] Navigation based on detected vehicle intervals Systems and methods for identifying vehicles and spacing between vehicles may enable navigating based on the detected spacing. Navigating in this manner may enable preemptive monitoring of movement within the detected spacing and reacting to such movement with shorter reaction times than traditional motion detection. Embodiments of the present disclosure described below relate to systems and methods for navigating based on detected spacing between vehicles.

[0265] 26 is an exemplary functional block diagram of memory 140 and / or 150 that may be stored / programmed with instructions for performing one or more operations in accordance with the disclosed embodiments. While the following is directed to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0266] 26 , memory 140 may store image analysis module 2602, vehicle identification module 2604, distance calculation module 2606, and navigation response module 2608. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 2602-2608 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Accordingly, any steps of the following processes may be performed by one or more processing devices.

[0267] In one embodiment, image analysis module 2602 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, perform image analysis of one or more images acquired by one of image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from a set of images with additional sensor information (e.g., information from radar, lidar, etc.) to perform the image analysis. As described below with respect to vehicle identification module 2604, image analysis module 2602 may include instructions for detecting vehicles using one or more features (e.g., front, rear, side, etc.).

[0268] In one embodiment, vehicle identification module 2604 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described above with respect to Figures 9-14, vehicle identification module 2604 may include instructions for determining bounding boxes for one or more vehicles.

[0269] In one embodiment, spacing calculation module 2606 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform an analysis of one or more images acquired by one of image capture devices 122, 124, and 126. As described with respect to Figures 27A and 27B, spacing calculation module 2606, in conjunction with vehicle identification module 2604, may include instructions for calculating one or more spacings between identified vehicles.

[0270] In one embodiment, navigation response module 2608 may store software executable by processing unit 110 to determine a desired navigation response based on data obtained from executing image analysis module 2602, vehicle identification module 2604, and / or separation calculation module 2606. For example, navigation response module 2608 may effect navigation changes according to method 2800 of FIG. 28, described below.

[0271] Additionally, any of the modules disclosed herein (e.g., modules 2602, 2604, and 2606) may implement techniques related to trained systems (such as neural networks or deep neural networks) or untrained systems.

[0272] Figure 27A shows a road 2702 from the perspective of a system according to a disclosed embodiment. Road 2702 may include multiple stopped vehicles, e.g., vehicle 2704 and vehicle 2706. As described below with respect to method 2800 of Figure 28, the system may identify bounding boxes for the sides of the stopped vehicles, e.g., side bounding box 2708 of vehicle 2704 and side bounding box 2710 of vehicle 2706. As described further below with respect to method 2800 of Figure 28, the system may determine the spacing between the identified bounding boxes using the front of one bounding box and the back of an adjacent bounding box, e.g., front 2712 of bounding box 2708 and back 2714 of bounding box 2710.

[0273] Figure 27B also shows road 2702 with vehicle 2704 and vehicle 2706 from the perspective of a system according to disclosed embodiments. Similar to Figure 27A, the system has identified side bounding box 2708 for vehicle 2704 and side bounding box 2710 for vehicle 2706. As shown in Figure 27B, hotspot 2716 has been identified based on the determined spacing between front 2712 of bounding box 2708 and rear 2714 of bounding box 2710. As described below with respect to method 2800 of Figure 28, the system can determine a navigation response for the host vehicle based on the identified hotspots.

[0274] 28 is a flowchart illustrating an example process 2800 for navigating based on detected inter-vehicle spacing, according to a disclosed embodiment. In step 2802, processing unit 110 may receive multiple images of the host vehicle's environment via data interface 128. For example, a camera included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture at least one image of an area in front of and / or to the side of the host vehicle and transmit them over a digital connection (e.g., USB, wireless, Bluetooth, etc.) to processing unit 110. Additional information from other sensors, such as radar, lidar, acoustic sensors, etc., may be used in combination with or in place of the multiple images.

[0275] In step 2804, processing unit 110 may analyze at least one of the plurality of images to identify at least two stopped vehicles. For example, identifying the stopped vehicles may further include associating at least one bounding box with a shape of a side of the stopped vehicle. For example, the analysis may be performed using the learned algorithm discussed above with respect to FIGS. 9 and 10.

[0276] In step 2806, processing unit 110 may determine a spacing between the identified vehicles. For example, processing unit 110 may scan at least one image from left to right (or right to left) to identify a right edge and an adjacent left edge. The identified right edge and the identified left edge may include the front of one bounding box and the back of another bounding box. The right edge and the left edge may form a gap pair from which a spacing may be calculated. In such an embodiment, the spacing may correspond to the distance between the front of one of the stopped vehicles and the back of the other stopped vehicle.

[0277] In other embodiments, the identified right edge and the identified left edge may include one bounding box on one side of the road and another bounding box on the other side of the road, and in such embodiments, the spacing may correspond to the distance between adjacent sides of the stopped vehicle.

[0278] In some embodiments, step 2806 may further include calculating a distance between the host vehicle and the determined interval. For example, processing unit 110 may calculate the distance based on the height and focal length of the host vehicle's image capture device (e.g., a camera). Based on known characteristics (such as height) of pedestrians or other objects, processing unit 110 may determine a shape within and / or near the calculated interval as a "hot spot" for the appearance of pedestrians or other objects. For example, the shape may be rectangular, oval, or other shape.

[0279] In step 2808, processing unit 110 may cause a navigational change of the host vehicle. For example, the navigational response may include a turn (as shown in FIG. 13 above), a lane shift (e.g., moving the host vehicle within a driving lane or changing the host vehicle's driving lane), a change in acceleration (e.g., slowing down the host vehicle), etc. In some embodiments, the at least one navigational change may be made by actuating at least one of the host vehicle's steering mechanism, brake, or accelerator.

[0280] Processing unit 110 may generate one or more navigation responses based on the calculated separation in step 2806. For example, processing unit 110 may determine that the calculated separation is sufficient to include a pedestrian. Based on this determination, processing unit may slow down the host vehicle and / or move the host vehicle away from the separation. In other words, a navigation change may occur if it is determined that the separation between two stopped vehicles is sufficient for a pedestrian to cross.

[0281] As a further example, processing unit 110 may determine that the calculated gap is sufficient to include the vehicle. Based on this determination, processing unit may slow down the host vehicle and / or move the host vehicle away from the gap. In other words, a navigation change may occur if it is determined that the gap between two stopped vehicles is sufficient for the target vehicle to cross.

[0282] As a further example, based on monitoring the hotspot, if a pedestrian or other object is identified within or near the hotspot, or if movement is detected within or near the hotspot, processing unit 110 may slow down the host vehicle and / or move the host vehicle away from the interval.

[0283] In some embodiments, processing unit 110 may detect a pedestrian in the gap between two stopped vehicles based on analyzing multiple images. For example, processing unit 110 may use the calculated gap and the expected height of the pedestrian to determine a location in the received image where the pedestrian's head may be expected to appear (e.g., hotspot 2716 in FIG. 27B ).

[0284] Pedestrian detection can be performed using a classifier trained for pedestrians, similar to the classifier trained for the vehicle side discussed above. In such an example, if a test point within a hotspot obtains a classifier score above an upper threshold, processing unit 110 may detect a definite pedestrian at that point. On the other hand, if a test point within a hotspot obtains a classifier score above a lower threshold but below an upper threshold, processing unit 110 may detect a suspect pedestrian at that point. The suspect pedestrian may be further tracked for movement toward a road, and if so, processing unit reclassifies the suspect pedestrian as a definite pedestrian. Such a detection method can improve upon conventional motion detection.

[0285] In some embodiments, at least a portion of the gap between two stopped vehicles may be obscured from the camera's view. In such embodiments, processing unit 110 may perform additional analysis to compensate for the ambiguity.

[0286] Processing unit 110 may also generate one or more navigational responses using data obtained from executing velocity and acceleration module 406. The multiple navigational responses may occur simultaneously, sequentially, or any combination thereof.

[0287] The above description has been presented for purposes of illustration. It is not exhaustive or limited to the precise form or embodiment disclosed. Modifications and adaptations will be apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments. Furthermore, while aspects of the disclosed embodiments are described as being stored in memory, those skilled in the art will appreciate that these aspects can also be stored on other types of computer-readable media, such as secondary storage devices, e.g., hard disks or CD-ROMs or other forms of RAM or ROM, USB media, DVDs, Blu-rays, 4K Ultra HD Blu-rays, or other optically driven media.

[0288] Computer programs based on the written descriptions and disclosed methods are within the skill of an experienced developer. The various programs or program modules can be created using any techniques known to those skilled in the art or can be designed in conjunction with existing software. For example, program sections or program modules can be designed in or with the .Net Framework, the .Net Compact Framework (and related languages ​​such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML / AJAX combinations, XML, or HTML including Java applets.

[0289] Furthermore, while exemplary embodiments are described herein, the scope of any embodiment includes equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations, and / or substitutions as would be understood by one of ordinary skill in the art based on this disclosure. Limitations in the claims should be construed broadly based on the language used in the claims, and not limited to the examples described herein or examples in the practice of this application. Examples should be construed as non-exclusive. Furthermore, the steps of the disclosed methods may be modified in any manner, including rearranging steps and / or inserting or deleting steps. Accordingly, it is intended that the specification and examples be considered merely as exemplary, with the true scope and spirit being indicated by the following claims and their full scope of equivalents. According to this specification, the matters described in the following items are also disclosed. [Item 1] 1. A system for navigating a host vehicle based on detecting a door opening event within an environment of the vehicle, comprising: receiving at least one image related to the environment of the host vehicle from an image capture device; analyzing the at least one image to identify a side of the parked vehicle; Identifying in the at least one image a first structural feature of the parked vehicle in a forward region of the side of the parked vehicle and a second structural feature of the parked vehicle in a rear region of the side of the parked vehicle; identifying a door edge of the parked vehicle in proximity to the first structural feature and the second structural feature within the at least one image; determining a change in an image characteristic of the door edge of the parked vehicle based on an analysis of one or more subsequent images received from the image capture device; Varying a navigation route of the host vehicle based at least in part on the change in the image characteristics of the door edge of the parked vehicle; and 1. A system including at least one processing device programmed to: [Item 2] Item 10. The system of item 1, wherein identifying the side of the parked vehicle includes associating at least one bounding box with a shape of the side of the parked vehicle. [Item 3] 2. The system of claim 1, wherein the first structural feature includes a front wheel component of the parked vehicle, and the second structural feature includes a rear wheel component of the parked vehicle. [Item 4] 2. The system of claim 1, wherein the vicinity of the first structural feature and the second structural feature includes an area between a front wheel component of the parked vehicle and a rear wheel component of the parked vehicle. [Item 5] 5. The system of claim 4, wherein the vicinity of the first wheel component and the second wheel component further includes an area above a rear wheel component of the parked vehicle. [Item 6] Item 10. The system of item 1, wherein the change in the image characteristics of the door edge includes the door edge widening. [Item 7] Item 10. The system of item 1, wherein the change in the image characteristic of the door edge includes the door edge moving away from a body of the vehicle. [Item 8] 2. The system of claim 1, wherein determining the change in the image characteristic of the door edge includes monitoring at least two images received from the image capture device for the appearance of vertical stripes in the vicinity of the first wheel component and the second wheel component. [Item 9] 9. The system of claim 8, wherein determining the change in the image characteristic of the door edge further comprises monitoring a spread of the vertical stripes. [Item 10] Item 10. The system of item 9, wherein a first edge of the vertical stripe is fixed along the body of the parked vehicle in the surveillance image, and a second edge of the vertical stripe moves toward a front area of ​​the parked vehicle in the surveillance image. [Item 11] Item 11. The system of item 10, wherein the at least one processing device is further configured to determine a width of the vertical stripes. [Item 12] Item 12. The system of item 11, wherein the at least one processing device is further configured to determine a distance the door edge extends away from the parked vehicle based on the determined width. [Item 13] Item 13. The system of item 12, wherein the at least one processing device is further configured to determine a lateral safety distance for the host vehicle based on the determined distance the door edge extends away from the parked vehicle. [Item 14] Item 14. The system of item 13, wherein altering the navigation path of the host vehicle is based on the determined lateral safety distance. [Item 15] 1. A method for navigating a host vehicle based on detecting a door opening event within an environment of the vehicle, comprising: receiving at least one image related to the environment of the host vehicle from an image capture device; analyzing the at least one image to identify a side of the parked vehicle; Identifying in the at least one image a first structural feature of the parked vehicle in a forward region of the side of the parked vehicle and a second structural feature of the parked vehicle in a rear region of the side of the parked vehicle; identifying a door edge of the parked vehicle in proximity to the first structural feature and the second structural feature within the at least one image; determining a change in an image characteristic of the door edge of the parked vehicle based on an analysis of one or more subsequent images received from the image capture device; Varying a navigation route of the host vehicle based at least in part on the change in the image characteristics of the door edge of the parked vehicle; and A method comprising: [Item 16] Item 16. The method of item 15, wherein determining the change in the image characteristic of the door edge includes monitoring at least two images received from the image capture device for the appearance of vertical stripes in the vicinity of the first structural feature and the second structural feature. [Item 17] Item 17. The method of item 16, wherein determining the change in the image characteristic of the door edge further comprises monitoring the spread of the vertical stripes. [Item 18] determining the width of the vertical stripes; determining a distance that the door edge extends away from the parked vehicle based on the determined width; Item 18. The method of item 17, further comprising: [Item 19] 19. The method of claim 18, further comprising determining a lateral safety distance for the host vehicle based on the determined distance the door edge extends away from the parked vehicle, and altering the navigation path of the host vehicle is based on the determined lateral safety distance. [Item 20] When executed by at least one processing device, receiving at least one image related to an environment of the host vehicle from an image capture device; analyzing the at least one image to identify a side of the parked vehicle; Identifying in the at least one image a first structural feature of the parked vehicle in a forward region of the side of the parked vehicle and a second structural feature of the parked vehicle in a rear region of the side of the parked vehicle; identifying a door edge of the parked vehicle in proximity to the first structural feature and the second structural feature within the at least one image; determining a change in an image characteristic of the door edge of the parked vehicle based on an analysis of one or more subsequent images received from the image capture device; Varying a navigation route of the host vehicle based at least in part on the change in the image characteristics of the door edge of the parked vehicle; and A non-transitory computer-readable medium storing instructions for performing a method including: [Item 21] 1. A system for navigating a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is traveling, comprising: receiving a plurality of images relating to an environment of the host vehicle from an image capture device; analyzing at least one of the plurality of images to identify the target vehicle and at least one wheel component on a side of the target vehicle; analyzing a region including the at least one wheel component of the target vehicle in at least two of the plurality of images to identify movement associated with the at least one wheel component of the target vehicle; causing at least one navigation change of the host vehicle based on the identified movement associated with the at least one wheel component of the target vehicle; and 1. A system including at least one processing device programmed to: [Item 22] 22. The system of claim 21, wherein the at least one navigation change includes a change in heading of the host vehicle. [Item 23] 22. The system of claim 21, wherein the at least one navigation change includes applying brakes to the host vehicle. [Item 24] 22. The system of claim 21, wherein identifying the target vehicle includes associating at least one bounding box with the shape of the target vehicle. [Item 25] 22. The system of claim 21, wherein the region including the at least one wheel component of the target vehicle includes a region adjacent to a road surface. [Item 26] Item 22. The system of item 21, wherein identifying movement associated with the at least one wheel component includes monitoring at least two of the plurality of images for signs of rotation of the at least one wheel component. [Item 27] 22. The system of claim 21, wherein identifying movement associated with the at least one tire / wheel component includes identifying at least one feature associated with the at least one wheel component and identifying an indicator of a position change of the at least one feature. [Item 28] 28. The system of claim 27, wherein the at least one processing device is further configured to determine a speed at which the target vehicle is traveling based on the position change of the at least one feature associated with the at least one wheel component. [Item 29] Item 29. The system of item 28, wherein the at least one processing device is further configured to determine a lateral safe distance for the vehicle based on the speed of the target vehicle. [Item 30] 30. The system of claim 29, wherein the at least one navigation change of the target vehicle is based on the determined lateral safe distance. [Item 31] 22. The system of claim 21, wherein the at least one wheel component includes at least one of a tire, a hubcap, or a wheel structure. [Item 32] 1. A method for navigating a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is traveling, comprising: receiving a plurality of images relating to an environment of the host vehicle from an image capture device; analyzing at least one of the plurality of images to identify the target vehicle and at least one wheel component on a side of the target vehicle; analyzing a region including the at least one wheel component of the target vehicle in at least two of the plurality of images to identify movement associated with the at least one wheel component of the target vehicle; causing at least one navigation change of the host vehicle based on the identified movement associated with the at least one wheel component of the target vehicle; and A method comprising: [Item 33] Item 33. The method of item 32, wherein identifying the target vehicle includes associating at least one bounding box with the shape of the target vehicle. [Item 34] Item 33. The method of item 32, wherein identifying movement associated with the at least one wheel component includes monitoring at least two of the plurality of images for signs of rotation of the at least one wheel component. [Item 35] 33. The method of claim 32, wherein identifying movement associated with the at least one tire / wheel component includes identifying at least one feature associated with the at least one wheel component and identifying an indicator of a position change of the at least one feature. [Item 36] determining a speed at which the target vehicle is traveling based on the change in position of the at least one feature associated with the at least one wheel component; determining a lateral safe distance for the target vehicle based on the speed of the target vehicle; and Item 36. The method of item 35, further comprising: [Item 37] When executed by at least one processing device, receiving a plurality of images relating to an environment of a host vehicle from an image capture device; analyzing at least one of the plurality of images to identify a target vehicle and at least one wheel component on a side of the target vehicle; analyzing a region including the at least one wheel component of the target vehicle in at least two of the plurality of images to identify movement associated with the at least one wheel component of the target vehicle; causing at least one navigation change of the host vehicle based on the identified movement associated with the at least one wheel component of the target vehicle; and A non-transitory computer-readable medium storing instructions for performing a method including: [Item 38] Item 38. The non-transitory computer-readable medium of item 37, wherein identifying the target vehicle includes associating at least one bounding box with a shape of the target vehicle. [Item 39] Item 38. The non-transitory computer-readable medium of item 37, wherein identifying movement associated with the at least one tire / wheel component includes identifying at least one feature associated with the at least one wheel component and identifying an indicator of a position change of the at least one feature. [Item 40] determining a speed at which the target vehicle is traveling based on the change in position of the at least one feature associated with the at least one wheel component; determining a lateral safe distance for the target vehicle based on the speed of the target vehicle; and Item 39. The non-transitory computer-readable medium of item 38, further storing instructions for performing [Item 41] 1. A system for detecting whether a road on which a host vehicle is traveling is a one-way road, comprising: receiving at least one image related to an environment of the host vehicle from an image capture device; identifying a first plurality of vehicles on a first side of the road along which the host vehicle is traveling based on analysis of the at least one image; identifying a second plurality of vehicles on a second side of the road along which the host vehicle is traveling based on analysis of the at least one image; and determining a first forward direction associated with the first plurality of vehicles; determining a second forward direction associated with the second plurality of vehicles; causing at least one navigation change of the host vehicle when both the first forward direction and the second forward direction are opposite to a direction of travel of the host vehicle; 1. A system including at least one processing device programmed to: [Item 42] Item 42. The system of item 41, wherein the at least one processing device is further programmed to identify a side of at least one of the first plurality of vehicles or at least one of the second plurality of vehicles based on analysis of the at least one image. [Item 43] Item 43. The system of item 42, wherein the identified side is the right side. [Item 44] Item 43. The system of item 42, wherein the identified side is the left side. [Item 45] Item 43. The system of item 42, wherein the at least one processing device is further programmed to identify the side based on at least two characteristics associated with at least one of the first plurality of vehicles or at least one of the second plurality of vehicles. [Item 46] Item 42. The system of item 41, wherein the at least one navigation change includes applying brakes to the host vehicle. [Item 47] 1. A method for detecting whether a road on which a host vehicle is traveling is a one-way road, comprising: receiving at least one image related to an environment of the host vehicle from an image capture device; identifying a first plurality of vehicles on a first side of the road along which the host vehicle is traveling based on analysis of the at least one image; identifying a second plurality of vehicles on a second side of the road along which the host vehicle is traveling based on analysis of the at least one image; and determining a first forward direction associated with the first plurality of vehicles; determining a second forward direction associated with the second plurality of vehicles; causing at least one navigation change of the host vehicle when both the first forward direction and the second forward direction are opposite to a direction of travel of the host vehicle; A method comprising: [Item 48] Item 48. The method of item 47, further comprising identifying a side of at least one of the first plurality of vehicles or at least one of the second plurality of vehicles based on an analysis of at least one of the plurality of images. [Item 49] Item 49. The method of item 48, wherein the identified side is the right side. [Item 50] Item 49. The method of item 48, wherein the identified side is the left side. [Item 51] 49. The method of claim 48, further comprising identifying the side based on at least two characteristics associated with at least one of the first plurality of vehicles or at least one of the second plurality of vehicles. [Item 52] 48. The method of claim 47, wherein the at least one navigation change includes applying the brakes of the host vehicle. [Item 53] When executed by at least one processing device, receiving at least one image related to an environment of the host vehicle from an image capture device; identifying a first plurality of vehicles on a first side of a road along which the host vehicle is traveling based on analysis of the at least one image; identifying a second plurality of vehicles on a second side of the road along which the host vehicle is traveling based on analysis of the at least one image; and determining a first forward direction associated with the first plurality of vehicles; determining a second forward direction associated with the second plurality of vehicles; causing at least one navigation change of the host vehicle when both the first forward direction and the second forward direction are opposite to a direction of travel of the host vehicle; A non-transitory computer-readable medium storing instructions for performing a method including: [Item 54] Item 54. The non-transitory computer-readable medium of Item 53, further storing instructions for identifying a side of at least one of the first plurality of vehicles or at least one of the second plurality of vehicles based on an analysis of at least one of the plurality of images. [Item 55] 1. A system for navigating a host vehicle, comprising: receiving navigation instructions for navigating the host vehicle from a first road along which the host vehicle is traveling to a second road; receiving at least one image related to the second road environment from an image capture device; identifying a first plurality of vehicles on a first side of the second road based on analysis of the at least one image; identifying a second plurality of vehicles on a second side of the second road based on analysis of the at least one image; determining a first forward direction associated with the first plurality of vehicles; determining a second forward direction associated with the second plurality of vehicles; determining that both the first forward direction and the second forward direction are opposite to a direction of travel in which the host vehicle will travel if the host vehicle enters the second road; interrupting the navigation instructions in response to the determination that both the first forward direction and the second forward direction are opposite to the direction of travel in which the host vehicle will travel if the host vehicle is navigated onto the second road; and 1. A system including at least one processing device programmed to: [Item 56] 56. The system of claim 55, wherein the at least one processing device is further programmed to receive an override instruction to restore the interrupted navigation instruction. [Item 57] Item 57. The system of item 56, wherein the override command is initiated based on manual confirmation received from a person within the host vehicle. [Item 58] Item 57. The system of item 56, wherein the override command is initiated based on accessing map data. [Item 59] Item 57. The system of item 56, wherein the override instruction is initiated based on crowd-sourced data related to the direction of travel on the second road. [Item 60] 1. A system for determining a predicted state of a parked vehicle within an environment of a host vehicle, comprising: an image capture device; an infrared image capture device; at least one processing device, receiving a plurality of images from the image capture device relating to the environment of the host vehicle; analyzing at least one of the plurality of images to identify the parked vehicle; analyzing at least two of the plurality of images to identify a change in illumination condition of at least one light associated with the parked vehicle; receiving at least one thermal image of the parked vehicle from the infrared image capture device; determining the predicted state of the parked vehicle based on the change in lighting conditions and an analysis of the at least one thermal image; generating at least one navigation response by the host vehicle based on the predicted state of the parked vehicle; and at least one processing device programmed to perform A system including: [Item 61] Item 61. The system of item 60, wherein the change in the illumination state of the at least one light associated with the parked vehicle includes a change from an unlit state to an lit state. [Item 62] Item 62. The system of item 61, wherein the predicted state of the parked vehicle includes an indication that the engine of the parked vehicle has been started. [Item 63] Item 61. The system of item 60, wherein the change in the illumination state of the at least one light associated with the parked vehicle includes a change from a lit state to an unlit state. [Item 64] Item 64. The system of item 63, wherein the predicted state of the parked vehicle includes an indication that the parked vehicle's engine has been turned off. [Item 65] Item 61. The system of item 60, wherein the analysis of the at least one thermal image includes determining a temperature value of an engine area of ​​the parked vehicle. [Item 66] Item 66. The system of item 65, wherein the predicted state of the parked vehicle is determined based at least in part on a comparison of the temperature value to a threshold value. [Item 67] Item 67. The system of item 66, wherein if the temperature value is below a threshold, the predicted state of the parked vehicle includes an indication that the parked vehicle is not expected to move within a predetermined period of time. [Item 68] Item 61. The system of item 60, wherein the analysis of the at least one thermal image includes determining a first temperature value of an engine area of ​​the parked vehicle and a second temperature value of at least one wheel component of the parked vehicle. [Item 69] Item 69. The system of item 68, wherein the predicted state of the parked vehicle is determined based on a comparison of the first temperature value to a first threshold and a comparison of the second temperature value to a second threshold. [Item 70] Item 69. The system of item 68, wherein if the first temperature value is above a first threshold and the second temperature value is below a second threshold, the predicted state of the parked vehicle includes an indication that the parked vehicle is expected to move within a predetermined period of time. [Item 71] Item 69. The system of item 68, wherein if the first temperature value exceeds a first threshold and the second temperature value exceeds a second threshold, the predicted state of the parked vehicle includes an indication that a door of the parked vehicle is expected to open within a predetermined period of time. [Item 72] Item 61. The system of item 60, wherein the predicted state of the parked vehicle includes at least one of an indication that the parked vehicle is not expected to move within a predetermined period of time, an indication that the parked vehicle is expected to move within a predetermined period of time, or an indication that a door of the parked vehicle is expected to open within a predetermined period of time. [Item 73] the at least one processing device registering at least one of the plurality of images with the at least one thermal image; identifying at least one of an engine region or at least one wheel component region of the parked vehicle within the registered at least one thermal image; Item 61. The system of item 60, further programmed to: [Item 74] 1. A method for determining a predicted state of a parked vehicle within an environment of a host vehicle, comprising: receiving a plurality of images relating to the environment of the host vehicle from an image capture device; analyzing at least one of the plurality of images to identify the parked vehicle; analyzing at least two of the plurality of images to identify a change in illumination condition of at least one light associated with the parked vehicle; receiving at least one thermal image of the parked vehicle from an infrared image capture device; determining the predicted state of the parked vehicle based on the change in lighting conditions and an analysis of the at least one thermal image; generating at least one navigation response by the host vehicle based on the predicted state of the parked vehicle; and A method comprising: [Item 75] The analysis of the at least one thermal image includes determining a temperature value of an engine area of ​​the parked vehicle; and Item 75. The method of item 74, wherein if the temperature value is below a threshold, the predicted state of the parked vehicle includes an indication that the parked vehicle is not expected to move within a predetermined period of time. [Item 76] The analysis of the at least one thermal image includes determining a first temperature value of an engine area of ​​the parked vehicle and a second temperature value of at least one wheel component of the parked vehicle; and Item 75. The method of item 74, wherein if the first temperature value is above a first threshold and the second temperature value is below a second threshold, the predicted state of the parked vehicle includes an indication that the parked vehicle is expected to move within a predetermined period of time. [Item 77] The analysis of the at least one thermal image includes determining a first temperature value of an engine area of ​​the parked vehicle and a second temperature value of at least one wheel component of the parked vehicle; and Item 75. The method of item 74, wherein if the first temperature value is above a first threshold and the second temperature value is above a second threshold, the predicted state of the parked vehicle includes an indication that a door of the parked vehicle is expected to open within a predetermined period of time. [Item 78] registering at least one of the plurality of images with the at least one thermal image; identifying at least one of an engine region or at least one wheel component region of the parked vehicle within the registered at least one thermal image; 75. The method of claim 74, further comprising: [Item 79] When executed by at least one processing device, receiving a plurality of images relating to an environment of a host vehicle from an image capture device; analyzing at least one of the plurality of images to identify a parked vehicle; analyzing at least two of the plurality of images to identify a change in illumination condition of at least one light associated with the parked vehicle; receiving at least one thermal image of the parked vehicle from an infrared image capture device; determining a predicted state of the parked vehicle based on the change in lighting conditions and an analysis of the at least one thermal image; generating at least one navigation response by the host vehicle based on the predicted state of the parked vehicle; and A non-transitory computer-readable medium storing instructions for performing a method including: [Item 80] 1. A system for determining a predicted state of a parked vehicle within an environment of a host vehicle, comprising: an image capture device; at least one processing device, receiving a plurality of images from the image capture device relating to the environment of the host vehicle; analyzing at least one of the plurality of images to identify the parked vehicle; analyzing at least two of the plurality of images to identify a change in illumination condition of at least one light associated with the parked vehicle; determining the predicted state of the parked vehicle based on the change in lighting conditions; generating at least one navigation response by the host vehicle based on the predicted state of the parked vehicle; and at least one processing device programmed to perform A system including: [Item 81] Item 81. The system of item 80, wherein the change in the illumination state of the at least one light associated with the parked vehicle includes a change from an unlit state to an lit state. [Item 82] Item 82. The system of item 81, wherein the predicted state of the parked vehicle includes an indication that the engine of the parked vehicle has been started. [Item 83] Item 81. The system of item 80, wherein the change in the illumination state of the at least one light associated with the parked vehicle includes a change from a lit state to an unlit state. [Item 84] Item 84. The system of item 83, wherein the predicted state of the parked vehicle includes an indication that the engine of the parked vehicle has been turned off. [Item 85] 1. A navigation system for a host vehicle, comprising: receiving a plurality of images from a camera representing an environment of the host vehicle; analyzing at least one of the plurality of images to identify at least two stopped vehicles; determining a spacing between the two stopped vehicles based on analyzing the at least one of the plurality of images; causing at least one navigation change for the host vehicle based on the determined separation between the two stopped vehicles; and 1. A system including at least one processing device programmed to: [Item 86] Item 86. The system of item 85, wherein the at least one navigation change includes slowing down the host vehicle. [Item 87] Item 86. The system of item 85, wherein the at least one navigation change includes moving the host vehicle within a driving lane. [Item 88] Item 86. The system of item 85, wherein the at least one navigation change includes changing a driving lane of the host vehicle. [Item 89] Item 86. The system of item 85, wherein effecting the at least one navigation change is performed by actuating at least one of the host vehicle's steering mechanism, brake, or accelerator. [Item 90] Item 86. The system of item 85, wherein the at least one navigation change of the host vehicle occurs when it is determined that the gap between the two stopped vehicles is sufficient for a pedestrian to cross. [Item 91] Item 86. The system of item 85, wherein the at least one navigation change of the host vehicle occurs when it is determined that the gap between the two stopped vehicles is sufficient for a target vehicle to cross. [Item 92] Item 86. The system of item 85, wherein the spacing corresponds to the distance between adjacent sides of the stopped vehicle. [Item 93] Item 86. The system of item 85, wherein the spacing corresponds to the distance between the front of one of the stopped vehicles and the rear of another stopped vehicle. [Item 94] Item 86. The system of item 85, wherein the at least one processing device is further programmed to detect a pedestrian within the gap between the two stopped vehicles based on analysis of the plurality of images. [Item 95] Item 86. The system of item 85, wherein at least a portion of the gap between the two stopped vehicles is hidden from the camera's view. [Item 96] 1. A method for navigating a host vehicle, comprising: receiving a plurality of images from a camera representing an environment of the host vehicle; analyzing at least one of the plurality of images to identify at least two stopped vehicles; determining a spacing between the two stopped vehicles based on analyzing the at least one of the plurality of images; causing at least one navigation change for the host vehicle based on the determined separation between the two stopped vehicles; and A method comprising: [Item 97] Item 97. The method of item 96, wherein causing at least one navigation change in the host vehicle includes slowing down the host vehicle. [Item 98] Item 97. The method of item 96, wherein causing at least one navigation change in the host vehicle includes moving the host vehicle within a driving lane. [Item 99] Item 97. The method of item 96, wherein causing at least one navigation change of the host vehicle includes changing a driving lane of the host vehicle. [Item 100] 97. The method of claim 96, wherein effecting the at least one navigation change includes actuating at least one of a steering mechanism, a brake, or an accelerator of the host vehicle. [Item 101] Item 97. The method of item 96, wherein the at least one navigation change of the host vehicle occurs when it is determined that the gap between the two stopped vehicles is sufficient for a pedestrian to cross. [Item 102] Item 97. The method of item 96, wherein the at least one navigation change of the host vehicle occurs when it is determined that the gap between the two stopped vehicles is sufficient for a target vehicle to cross. [Item 103] When executed by at least one processing device, receiving a plurality of images from a camera representing an environment of a host vehicle; analyzing at least one of the plurality of images to identify at least two stopped vehicles; determining a spacing between the two stopped vehicles based on analyzing the at least one of the plurality of images; causing at least one navigation change for the host vehicle based on the determined separation between the two stopped vehicles; and A non-transitory computer-readable medium storing instructions for performing a method including: [Item 104] Item 104. The non-transitory computer-readable medium of item 103, wherein causing at least one navigation change in the host vehicle includes at least one of slowing down the host vehicle, moving the host vehicle within a driving lane, or changing the driving lane of the host vehicle.

Claims

1. 1. A navigation system for an autonomously driving host vehicle, comprising: receiving a plurality of images from a camera representing an environment of the autonomously driving host vehicle; analyzing at least one of the plurality of images to identify at least two stopped vehicles; determining a spacing between the two stopped vehicles based on the analysis of the at least one of the plurality of images; determining whether the gap is wide enough for a target object to cross; Based on the determination of whether the determined gap between the two stopped vehicles is large enough for the target object to cross, if it is determined that the gap is large enough for the target object to cross, causing at least one navigation change in the autonomously driving host vehicle to move the autonomously driving host vehicle away from the gap; 1. A system comprising at least one processing device programmed to:

2. 2. The system of claim 1, wherein the at least one processing device is further configured to: slow down the autonomously driving host vehicle if it is determined that the gap determined between the two stopped vehicles is large enough for the target object to cross, based on the determination of whether the gap is large enough for the target object to cross.

3. The system of claim 1 or 2, wherein the at least one navigation change includes moving the autonomously driving host vehicle within a driving lane.

4. 4. The system of claim 1, wherein the at least one navigation change includes changing a driving lane of the autonomously driving host vehicle.

5. 5. The system of claim 1, wherein effecting the at least one navigation change is performed by actuating a steering mechanism of the autonomously driving host vehicle.

6. The system of claim 1 , wherein the target object is a pedestrian.

7. The system of claim 1 , wherein the target object is a target vehicle.

8. 8. A system according to claim 1, wherein the spacing corresponds to the distance between a side of one of the stopped vehicles and an adjacent side of another stopped vehicle.

9. 8. A system according to claim 1, wherein the spacing corresponds to the distance between the front of one of the stopped vehicles and the rear of another of the stopped vehicles.

10. 10. The system of claim 1, wherein the at least one processing device is further programmed to detect a pedestrian within the gap between the two stopped vehicles based on analysis of the plurality of images.

11. 11. The system of claim 1, wherein at least a portion of the gap between the two stopped vehicles is hidden from the view of the camera.

12. 1. A method for navigating an autonomously driving host vehicle, comprising: receiving a plurality of images from a camera representing an environment of the autonomously driving host vehicle; analyzing at least one of the plurality of images to identify at least two stopped vehicles; determining a spacing between the two stopped vehicles based on the analysis of the at least one of the plurality of images; determining whether the gap is wide enough for a target object to cross; Based on the determination of whether the determined gap between the two stopped vehicles is large enough for the target object to cross, if it is determined that the gap is large enough for the target object to cross, causing at least one navigation change in the autonomously driving host vehicle to move the autonomously driving host vehicle away from the gap; A method for providing

13. The method of claim 12, further comprising slowing down the autonomously driving host vehicle if it is determined that the gap between the two stopped vehicles is wide enough for the target object to cross, based on the determination of whether the gap is wide enough for the target object to cross.

14. 14. The method of claim 12 or 13, wherein causing at least one navigation change in the autonomously driving host vehicle includes moving the autonomously driving host vehicle within a driving lane.

15. 15. The method of any one of claims 12 to 14, wherein effecting at least one navigation change in the host vehicle that is driving autonomously includes changing a driving lane of the host vehicle that is driving autonomously.

16. 16. The method of claim 12, wherein effecting the at least one navigation change comprises actuating a steering mechanism of the autonomously driving host vehicle.

17. The method of any one of claims 12 to 16, wherein the target object is a pedestrian.

18. The method of any one of claims 12 to 16, wherein the target object is a target vehicle.

19. receiving a plurality of images from a camera representing an environment of a host vehicle that is operating autonomously; analyzing at least one of the plurality of images to identify at least two stopped vehicles; determining a spacing between the two stopped vehicles based on the analysis of the at least one of the plurality of images; determining whether the gap is wide enough for a target object to cross; Based on the determination of whether the determined gap between the two stopped vehicles is large enough for the target object to cross, if it is determined that the gap is large enough for the target object to cross, causing at least one navigation change in the autonomously driving host vehicle to move the autonomously driving host vehicle away from the gap; A program that causes a computer to execute the following.

20. 20. The program of claim 19, wherein causing at least one navigation change in the autonomously driving host vehicle includes at least one of moving the autonomously driving host vehicle within a driving lane or changing a driving lane of the autonomously driving host vehicle.

Images