System and method for identifying potential communication impairments

Abstract

Systems and methods for vehicle navigation are provided. In one implementation, a navigation system for a vehicle can include at least one processor. The at least one processor can be programmed to obtain a route from a location of the vehicle to a destination; receive signal information indicative of a quality characteristic of a signal associated with at least one location along the route; determine a quality parameter of at least one operational characteristic associated with a communication system; and determine at least one modification to the route for the vehicle based on the signal information and the quality parameter.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 891,652, filed August 26, 2019. The aforementioned application is incorporated herein by reference in its entirety. BACKGROUND TECHNICAL FIELD

[0004] The present disclosure relates generally to autonomous vehicle navigation. BACKGROUND

[0006] As technology continues to advance, the goal of a fully autonomous vehicle capable of navigating on roadways is within reach. An autonomous vehicle can need to take into account a variety of factors and make appropriate decisions based on those factors to safely and accurately reach an intended destination. For example, an autonomous vehicle can need to process and interpret visual information (e.g., information captured from a camera) and can also use information obtained from other sources (e.g., from a GPS device, a speed sensor, an accelerometer, a suspension sensor, etc.). At the same time, to navigate to a destination, an autonomous vehicle can also need to identify its position within a particular roadway (e.g., a particular lane within a multi-lane roadway), navigate alongside other vehicles, avoid obstacles and pedestrians, observe traffic signals and signs, and travel from one road to another at appropriate intersections or grade-separated roadways. Utilizing and interpreting the large amount of information collected by an autonomous vehicle as it travels to its destination presents numerous design challenges. The sheer amount of data (e.g., captured image data, map data, GPS data, sensor data, etc.) that an autonomous vehicle can need to analyze, access, and / or store presents challenges that can in fact limit or even adversely affect autonomous navigation. Moreover, if an autonomous vehicle relies on traditional map technology for navigation, the sheer amount of data needed to store and update the map presents a daunting challenge.

[0007] Traditional vehicle navigation systems rely on data received over a wireless communication network (e.g., a cellular network), which is typically streamed to the vehicle while the vehicle is in motion. As a result, many vehicles depend on a reliable data connection to receive data during navigation. A weak or non-existent connection can result in a lack of navigation data at the vehicle. In the case of a vehicle that is autonomous or semi-autonomous, such an interruption can result in a suspension of autonomous or semi-autonomous operation of the vehicle.

[0008] The disclosed embodiments can address one or more of the problems described above. SUMMARY

[0009] Embodiments consistent with the present disclosure provide systems and methods for vehicle navigation. Disclosed embodiments can receive signal information to determine a route for a vehicle or a plurality of vehicles. For example, consistent with disclosed embodiments, a disclosed system can include a navigation system that can modify a route for a vehicle based on signal information received from a remote source. The disclosed system can provide a navigation response based on, for example, an analysis of signal information regarding a quality parameter.

[0010] In embodiments, a navigation system for a vehicle can include at least one processor. The at least one processor can be programmed to: obtain a route from a location of the vehicle to a destination; receive signal information indicative of a quality characteristic of a signal associated with at least one location along the route; determine a quality parameter of at least one operational characteristic associated with a communication system; and determine at least one modification to the route for the vehicle based on the signal information and the quality parameter.

[0011] In embodiments, a system for guiding autonomous vehicles along a route can include at least one processor. The at least one processor can be programmed to: receive a first location of a first autonomous vehicle; receive a second location of a second autonomous vehicle; receive a destination, wherein the destination is closer to the first location of the first autonomous vehicle than to the second location of the second autonomous vehicle; determine a first route from the first location to the destination and a second route from the second location to the destination; receive first signal information indicative of a first quality characteristic of a first signal associated with at least one location along the first route; receive second signal information indicative of a second quality characteristic of a second signal associated with at least one location along the second route; receive a quality parameter of at least one operational characteristic associated with a first communication system of the first autonomous vehicle and a second communication system of the second autonomous vehicle; determine, based on the first location, the second location, and the quality parameter, whether to guide the first autonomous vehicle or the second autonomous vehicle along the route to the destination; and transmit instruction information configured to cause the second autonomous vehicle to travel from the second location to the destination.

[0012] Consistent with other disclosed embodiments, a non-transitory computer- readable storage medium can store program instructions for execution by at least one processing device and to perform any of the methods described herein.

[0013] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings:

[0015] Figure 1 is a diagrammatic representation of an example system consistent with the disclosed embodiments.

[0016] Figure 2A is a diagrammatic side view representation of an example vehicle including a system consistent with the disclosed embodiments;

[0017] Figure 2B is a diagrammatic representation of a camera mount configured to be positioned behind a rearview mirror and against a vehicle windshield consistent with the disclosed embodiments; Figure 2A is a diagrammatic top view representation of the vehicle and system shown in

[0018] Figure 2C is a diagrammatic top view representation of another embodiment of a vehicle including a system consistent with the disclosed embodiments;

[0019] Figure 2D is a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments;

[0020] Figure 2E is a diagrammatic top view representation of yet another embodiment of a vehicle including a system consistent with the disclosed embodiments;

[0021] Figure 2F is a diagrammatic representation of an example vehicle control system consistent with the disclosed embodiments.

[0022] Figure 3A is a diagrammatic representation of an interior of a vehicle including a rearview mirror and a user interface for a vehicle imaging system consistent with the disclosed embodiments;

[0023] Figure 3B is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against a vehicle windshield consistent with the disclosed embodiments.

[0024] Figure 3C is an illustration of a camera mount shown in Figure 3B from a different perspective consistent with the disclosed embodiments.

[0025] Figure 3D is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against a vehicle windshield consistent with the disclosed embodiments.

[0026] Figure 4 is an example block diagram of a memory configured to store instructions for performing one or more operations consistent with the disclosed embodiments.

[0027] Figure 5A is a flowchart showing a process for causing one or more navigational responses based on a monocular image consistent with the disclosed embodiments.

[0028] Figure 5B FIG. 1 is a flowchart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a set of images, consistent with disclosed embodiments.

[0029] Figure 5C FIG. 2 is a flowchart illustrating an exemplary process for detecting road markings and / or lane geometry information in a set of images, consistent with disclosed embodiments.

[0030] Figure 5D FIG. 3 is a flowchart illustrating an exemplary process for detecting traffic signals in a set of images, consistent with disclosed embodiments.

[0031] Figure 5E FIG. 4 is a flowchart illustrating an exemplary process for causing one or more navigational responses based on vehicle path, consistent with disclosed embodiments.

[0032] Figure 5F FIG. 5 is a flowchart illustrating an exemplary process for determining whether a preceding vehicle is changing lanes, consistent with disclosed embodiments.

[0033] Figure 6 FIG. 6 is a flowchart illustrating an exemplary process for causing one or more navigational responses based on stereoscopic image analysis, consistent with disclosed embodiments.

[0034] Figure 7 FIG. 7 is a flowchart illustrating an exemplary process for causing one or more navigational responses based on analysis of three sets of images, consistent with disclosed embodiments.

[0035] Figure 8 FIG. 8 illustrates a sparse map for providing autonomous vehicle navigation, consistent with disclosed embodiments.

[0036] Figure 9A FIG. 9 illustrates a polynomial representation of a portion of a road segment, consistent with disclosed embodiments.

[0037] Figure 9B FIG. 10 illustrates a curve in three-dimensional space representing a target trajectory for a vehicle in a sparse map for a particular road segment, consistent with disclosed embodiments.

[0038] Figure 10 FIG. 11 illustrates an example landmark that can be included in a sparse map, consistent with disclosed embodiments.

[0039] Figure 11A FIG. 12 illustrates a polynomial representation of a trajectory, consistent with disclosed embodiments.

[0040] Figure 11B and Figure 11CTarget trajectories along a multi-lane road are shown consistent with the disclosed embodiments.

[0041] Figure 11D An example road signature profile is shown consistent with the disclosed embodiments.

[0042] Figure 12 is a schematic diagram of a system for autonomous vehicle navigation using crowd-sourced data received from multiple vehicles consistent with the disclosed embodiments.

[0043] Figure 13 An example autonomous vehicle road navigation model represented by multiple three-dimensional splines is shown consistent with the disclosed embodiments.

[0044] Figure 14 A map skeleton generated by combining location information from many drives is shown consistent with the disclosed embodiments.

[0045] Figure 15 An example of longitudinal alignment of two drives with example signs as landmarks is shown consistent with the disclosed embodiments.

[0046] Figure 16 An example of longitudinal alignment of many drives with example signs as landmarks is shown consistent with the disclosed embodiments.

[0047] Figure 17 is a schematic diagram of a system for generating data for a drive using a camera, a vehicle, and a server consistent with the disclosed embodiments.

[0048] Figure 18 is a schematic diagram of a system for crowd-sourcing sparse maps consistent with the disclosed embodiments.

[0049] Figure 19 is a flowchart showing an example process for generating a sparse map for autonomous vehicle navigation along a road segment consistent with the disclosed embodiments.

[0050] Figure 20 A block diagram of a server is shown consistent with the disclosed embodiments.

[0051] Figure 21 A block diagram of a memory is shown consistent with the disclosed embodiments.

[0052] Figure 22 A process to cluster vehicle trajectories associated with a vehicle is shown consistent with the disclosed embodiments.

[0053] Figure 23 A navigation system of a vehicle that can be used for autonomous navigation is shown consistent with the disclosed embodiments.

[0054] Figure 24A 、 24B FIGS. 24C and 24D illustrate example lane markings that can be detected consistent with the disclosed embodiments.

[0055] Figure 24E FIG. 25 illustrates example mapped lane markings consistent with the disclosed embodiments.

[0056] Figure 24F FIG. 26 illustrates example anomalies associated with detecting lane markings consistent with the disclosed embodiments.

[0057] Figure 25A FIG. 27 illustrates example images of a vehicle’s surroundings for navigation based on mapped lane markings consistent with the disclosed embodiments.

[0058] Figure 25B FIG. 28 illustrates correction of a vehicle’s lateral positioning based on mapped lane markings in a road navigation model consistent with the disclosed embodiments.

[0059] Figure 26A FIG. 29 is a flowchart illustrating an example process for mapped lane markings for autonomous vehicle navigation consistent with the disclosed embodiments.

[0060] Figure 26B FIG. 30 is a flowchart illustrating an example process for autonomously navigating a host vehicle along a road segment using mapped lane markings consistent with the disclosed embodiments.

[0061] Figure 27 FIG. 31 is a schematic diagram of a system for determining a route using signal information.

[0062] Figure 28 FIG. 32 is a schematic diagram of a system for determining a route between multiple vehicles using signal information.

[0063] Figure 29 FIG. 33 is a schematic diagram of a system for generating navigation information using signal information from vehicles.

[0064] Figure 30 FIG. 34 is a flowchart illustrating an example process for determining route information for vehicles.

[0065] Figure 31 FIG. 35 is a flowchart illustrating an example process for determining a route between vehicles. DETAILED DESCRIPTION

[0066] The following detailed description references the accompanying drawings. Same reference numerals in different drawings identify same or similar elements. Although several illustrative embodiments are described below, modifications, adaptations and other implementations are possible. For example, components illustrated in the figures can be rearranged, added, or omitted, and the illustrative methods described herein can be modified, rearranged, removed or added to. Therefore, the following detailed description is not intended to be limited to the disclosed embodiments and examples. Instead, the appropriate scope is defined only by the appended claims.

[0067] Overview of Autonomous Vehicles

[0068] As used throughout this disclosure, the term“autonomous vehicle” refers to a vehicle capable of effecting at least one navigational change without having driver input. A“navigational change” refers to one or more changes in steering, braking, or acceleration of the vehicle. To be autonomous, a vehicle need not be fully automated (e.g., operate completely without a driver or without driver input). Rather, autonomous vehicles include those that can operate under driver control during certain time periods and without driver control during other time periods. Autonomous vehicles can also include vehicles that control only some aspects of the vehicle’s navigation, such as steering (e.g., to maintain a vehicle course between vehicle lane constraints), but can leave other aspects to the driver (e.g., braking). In some cases, autonomous vehicles can handle some or all aspects of braking, speed control, and / or steering of the vehicle.

[0069] Because human drivers typically rely on visual cues and observations to control vehicles, the transportation infrastructure is accordingly established, with lane markings, traffic signs, and traffic signal lights all designed to provide visual information to drivers. In view of these design characteristics of the transportation infrastructure, autonomous vehicles can include cameras and processing units that analyze visual information captured from the vehicle’s environment. The visual information can include, for example, components of the transportation infrastructure that can be observed by a driver (e.g., lane markings, traffic signs, traffic lights, etc.) and other obstacles (e.g., other vehicles, pedestrians, debris, etc.). Additionally, autonomous vehicles can also use stored information, such as information that provides a model of the vehicle’s environment at the time of navigation. For example, a vehicle can use GPS data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data to provide information related to the vehicle’s environment as the vehicle is traveling, and the vehicle (and other vehicles) can use this information to position itself on the model.

[0070] In some embodiments of the present disclosure, an autonomous vehicle can use information obtained while navigating (e.g., from cameras, GPS devices, accelerometers, rate sensors, suspension sensors, etc.). In other embodiments, an autonomous vehicle can use information obtained from past navigations by the vehicle (or other vehicles) while navigating. In still other embodiments, an autonomous vehicle can use a combination of information obtained while navigating and information obtained from past navigations. The following sections provide an overview of a system consistent with the disclosed embodiments, followed by an overview of forward imaging systems and methods consistent with the system. Subsequent sections disclose systems and methods for constructing, using, and updating sparse maps for autonomous vehicle navigation.

[0071] Overview of System

[0072] Figure 1 is a block diagram representation of a system 100 consistent with the disclosed example embodiments. Depending on the requirements of a particular implementation, the system 100 can include various components. In some embodiments, the system 100 can include a processing unit 110, an image acquisition unit 120, a position sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. The processing unit 110 can include one or more processing devices. In some embodiments, the processing unit 110 can include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, depending on the requirements of a particular application, the image acquisition unit 120 can include any number of image acquisition devices and components. In some embodiments, the image acquisition unit 120 can include one or more image capture devices (e.g., cameras), such as image capture device 122, image capture device 124, and image capture device 126. The system 100 can also include a data interface 128 communicatively connecting the processing device 110 to the image acquisition device 120. For example, the data interface 128 can include any wireline and / or wireless link(s) for transferring image data acquired by the image acquisition device 120 to the processing unit 110.

[0073] The wireless transceiver 172 can include one or more devices configured to exchange transmissions over an air interface to one or more networks (e.g., cellular, the Internet, etc.) using radio frequencies, infrared frequencies, magnetic fields, or electric fields. The wireless transceiver 172 can use any known standard to transmit and / or receive data (e.g., Wi-Fi, Bluetooth®, ZigBee®, Z-Wave®, etc.). In some embodiments, the wireless transceiver 172 can include a cellular transceiver configured to communicate with a cellular network using any known cellular standard (e.g., GSM, CDMA, UMTS, LTE, 5G, etc.). In some embodiments, the wireless transceiver 172 can include a Wi-Fi transceiver configured to communicate with a Wi-Fi network using any known Wi-Fi standard (e.g., IEEE 802.11, etc.). In some embodiments, the wireless transceiver 172 can include a Bluetooth transceiver configured to communicate with a Bluetooth network using any known Bluetooth standard (e.g., Bluetooth Low Energy, etc.). In some embodiments, the wireless transceiver 172 can include a ZigBee transceiver configured to communicate with a ZigBee network using any known ZigBee standard (e.g., IEEE 802.15.4, etc.). In some embodiments, the wireless transceiver 172 can include a Z-Wave transceiver configured to communicate with a Z-Wave network using any known Z-Wave standard (e.g., IEEE 802.15.4, etc.). (Bluetooth Smart, 802.15.4, ZigBee, etc.). Such transmissions may include communication from a primary vehicle to one or more remotely located servers. Such transmissions may also include communication (one-way or two-way) between the primary vehicle and one or more target vehicles in the environment of the primary vehicle (e.g., for coordination of navigation of the primary vehicle in view of or in conjunction with the target vehicles in the environment of the primary vehicle), or even broadcast transmissions to unspecified recipients near the vehicle making the transmission.

[0074] Both application processor 180 and image processor 190 can include various types of processing devices. For example, either or both of application processor 180 and image processor 190 can include a microprocessor, preprocessor (e.g., an image preprocessor), graphics processing unit (GPU), central processing unit (CPU), support circuitry, digital signal processor, integrated circuit, memory, or any other type of device suitable for running applications and for image processing and analysis. In some embodiments, application processor 180 and / or image processor 190 can include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, etc. Various processing devices can be used, including, for example, those available from, such as... Processors obtained from manufacturers such as [list of manufacturers], or from [list of manufacturers such as [list of manufacturers] GPUs obtained from manufacturers, and various processing devices can include various architectures (e.g., x86 processors, etc.). wait).

[0075] In some embodiments, application processor 180 and / or image processor 190 may include components that can be accessed from... Any processor chip from the EyeQ series of processor chips obtained. These processor designs each include multiple processing units with local memory and instruction sets. Such processors may include video input for receiving image data from multiple image sensors, and may also include video output capabilities. In one example, It uses 90nm micrometer technology operating at 332MHz. The architecture consists of two floating-point hyper-threaded 32-bit RISC CPUs ( (core), five visual computing engines (VCE), and three vector microcode processors The system consists of a Denali 64-bit mobile DDR controller, 128-bit internal acoustic interconnect, dual 16-bit video input and 18-bit video output controllers, a 16-channel DMA, and several peripheral devices. The MIPS34K CPU manages five VCEs and three VMPs. TMand a second MIPS34K CPU and multi-channel DMA, and other peripherals. Five VCEs, three The MIPS34K CPU can perform the intensive visual computations required by multi-function bundled applications. In another example, as a third generation processor and six times more powerful than the EyeQ2 may be used in the disclosed embodiments. In other examples, and / or may be used in the disclosed embodiments. Of course, any newer or future EyeQ processing devices can also be used with the disclosed embodiments.

[0076] Any of the processing devices disclosed herein can be configured to perform certain functions. Configuring a processing device, such as any of the described EyeQ processors or other controllers or microprocessors, to perform certain functions can include programming computer-executable instructions and making those instructions available to the processing device to execute during operation of the processing device. In some embodiments, configuring a processing device can include programming the processing device directly with architectural instructions. For example, processing devices such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and similar processing devices can be configured using, for example, one or more hardware description languages (HDLs).

[0077] In other embodiments, configuring a processing device can include storing executable instructions on a memory accessible to the processing device during operation. For example, the processing device can access the memory during operation to obtain and execute the stored instructions. In either case, a processing device configured to perform the sensing, image analysis, and / or navigation functions disclosed herein represents a dedicated hardware-based system that controls multiple hardware-based components of a host vehicle.

[0078] While Figure 1 While two separate processing devices included in the processing unit 110 are depicted, more or fewer processing devices can be used. For example, in some embodiments, a single processing device can be used to implement the tasks of the application processor 180 and the image processor 190. In other embodiments, these tasks can be performed by more than two processing devices. Furthermore, in some embodiments, the system 100 can include one or more processing units 110 without including other components, such as the image acquisition unit 120.

[0079] The processing unit 110 can include various types of devices. For example, the processing unit 110 can include various devices such as a controller, an image preprocessor, a central processing unit (CPU), a graphics processing unit (GPU), support circuitry, a digital signal processor, an integrated circuit, a memory, or any other type of device for image processing and analysis. The image preprocessor can include a video processor for capturing, digitizing, and processing imagery from an image sensor. The CPU can include any number of microcontrollers or microprocessors. The GPU can also include any number of microcontrollers or microprocessors. The support circuitry can be any number of circuits generally known in the art including cache, power supply, clock, and input-output circuits. The memory can store software that, when executed by the processor, controls the operation of the system. The memory can include databases and image processing software. The memory can include any number of random access memories, read only memories, flash memories, disk drives, optical storage, tape storage, removable storage, and other types of storage. In one example, the memory can be separate from the processing unit 110. In another example, the memory can be integrated into the processing unit 110.

[0080] Each memory 140, 150 can include software instructions that, when executed by a processor (e.g., the application processor 180 and / or the image processor 190), can control the operation of various aspects of the system 100. These memory units can include various databases and image processing software, as well as trained systems such as neural networks or deep neural networks, for example. The memory units can include random access memory (RAM), read only memory (ROM), flash memory, disk drives, optical storage, tape storage, removable storage, and / or any other type of storage. In some embodiments, the memory units 140, 150 can be separate from the application processor 180 and / or the image processor 190. In other embodiments, these memory units can be integrated into the application processor 180 and / or the image processor 190.

[0081] The position sensor 130 can include any type of device suitable for determining a position associated with at least one component of the system 100. In some embodiments, the position sensor 130 can include a GPS receiver. Such a receiver can determine a user’s position and velocity by processing signals broadcast by global positioning system satellites. The position information from the position sensor 130 can be made available to the application processor 180 and / or the image processor 190.

[0082] In some embodiments, the system 100 can include components such as a speed sensor (e.g., a tachometer, a speedometer) for measuring the speed of the vehicle 200 and / or an accelerometer (single-axis or multi-axis) for measuring the acceleration of the vehicle 200.

[0083] User interface 170 can include any devices suitable for providing information to or receiving input from one or more users of system 100. In some embodiments, user interface 170 can include user input devices, including, for example, a touchscreen, a microphone, a keyboard, a pointing device, a trackwheel, a camera, a knob, a button, etc. With such input devices, a user can be able to input information or commands to system 100 by typing instructions or information, providing voice commands, using buttons, a pointer, or eye-tracking capabilities to select menu options on a screen, or by any other suitable technique for transmitting information to system 100.

[0084] User interface 170 can be equipped with one or more processing devices configured to provide information to or receive information from a user and process that information for use by, for example, application processor 180. In some embodiments, such processing devices can execute instructions for recognizing and tracking eye movements, receiving and interpreting voice commands, recognizing and interpreting touches and / or gestures made on a touchscreen, responding to keyboard inputs or menu selections, etc. In some embodiments, user interface 170 can include a display, a speaker, a haptic device, and / or any other devices for providing output information to a user.

[0085] Map database 160 can include any type of database for storing map data useful to system 100. In some embodiments, map database 160 can include data relating to the locations of various items, including roads, water features, geographic features, businesses, places of interest, restaurants, gas stations, etc., in a reference coordinate system. Map database 160 can store not only the locations of such items, but also descriptors related to those items, including, for example, names associated with any of the stored features. In some embodiments, map database 160 can be physically located with other components of system 100. Alternatively or additionally, map database 160 or portions thereof can be located remotely with respect to other components of system 100 (e.g., processing unit 110). In such embodiments, information from map database 160 can be downloaded over a wired or wireless data connection to a network (e.g., over a cellular network and / or the Internet, etc.). In some cases, map database 160 can store a sparse data model including a polynomial representation of certain road features (e.g., lane markings) or target trajectories for host vehicles. Systems and methods for generating such maps are discussed below with reference to Figures 8-19

[0086] ​Image capture devices 122, 124, and 126 can each include any type of device suitable for capturing at least one image from an environment. Further, any number of image capture devices can be used to gather images for input to the image processor. Some embodiments can include only a single image capture device, while other embodiments can include two, three, or even four or more image capture devices. Reference will be made below to Figures 2B-2E Image capture devices 122, 124, and 126 are further described.

[0087] System 100, or various components thereof, can be incorporated into a variety of different platforms. In some embodiments, system 100 can be included on a vehicle 200, as Figure 2A shown. For example, as discussed above with respect to Figure 1 vehicle 200 can be equipped with processing unit 110 and any of the other components of system 100. While in some embodiments, vehicle 200 can be equipped with only a single image capture device (e.g., a camera), in other embodiments, such as those discussed in connection with Figures 2B-2E multiple image capture devices can be used. For example, as Figure 2A shown, either of image capture devices 122 and 124 of vehicle 200 can be part of an ADAS (Advanced Driver Assistance System) imaging suite.

[0088] Image capture devices included on vehicle 200 as part of image gathering unit 120 can be positioned in any suitable location. In some embodiments, as Figures 2A-2E and Figures 3A-3C shown, image capture device 122 can be located near a rearview mirror. This location can provide a line of sight similar to that of a driver of vehicle 200, which can assist in determining what is and is not visible to the driver. Image capture device 122 can be positioned in any location near the rearview mirror, but placing image capture device 122 on the driver side of the rearview mirror can further assist in obtaining images representative of the field of view and / or line of sight of the driver.

[0089] Other positions for the image capture devices of the image acquisition unit 120 can also be used. For example, the image capture device 124 can be located on or in a bumper of the vehicle 200. Such a location can be particularly suitable for image capture devices having a wide field of view. The line of sight of an image capture device located in a bumper can be different from the line of sight of a driver, and thus, the bumper image capture device and the driver can not always see the same objects. Image capture devices (e.g., image capture devices 122, 124, and 126) can also be located in other positions. For example, an image capture device can be located on or in one or both of the side mirrors of the vehicle 200, on the roof of the vehicle 200, on the hood of the vehicle 200, on the trunk of the vehicle 200, on the side of the vehicle 200, mounted on any of the windows of the vehicle 200, positioned behind or in front of any of the windows of the vehicle 200, and mounted in or near a light on the front and / or back of the vehicle 200, etc.

[0090] In addition to the image capture devices, the vehicle 200 can also include various other components of the system 100. For example, the processing unit 110 can be included on the vehicle 200, integrated with or separate from an engine control unit (ECU) of the vehicle. The vehicle 200 can also be equipped with a position sensor 130 (such as a GPS receiver), and can also include a map database 160, as well as memory units 140 and 150.

[0091] As previously discussed, the wireless transceiver 172 can receive data over one or more networks (e.g., a cellular network, the Internet, etc.) and / or by one or more networks. For example, the wireless transceiver 172 can upload data collected by the system 100 to one or more servers, and download data from one or more servers. Via the wireless transceiver 172, the system 100 can receive, for example, periodic or on-demand updates to data stored in the map database 160, the memory 140, and / or the memory 150. Similarly, the wireless transceiver 172 can upload any data from the system 100 (e.g., images captured by the image acquisition unit 120, data received by the position sensor 130 or other sensors, vehicle control systems, etc.) to one or more servers and / or upload any data processed by the processing unit 110 to one or more servers.

[0092] The system 100 can upload data to a server (e.g., to the cloud) based on the privacy level setting. For example, the system 100 can implement a privacy level setting to regulate or limit the type of data (including metadata) sent to the server that can uniquely identify the vehicle and / or the driver / owner of the vehicle. Such settings can be set by the user via, for example, the wireless transceiver 172, initialized by factory default settings, or by data received by the wireless transceiver 172.

[0093] In some embodiments, the system 100 can upload data according to a“high” privacy level, and under the setting, the system 100 can transmit data (e.g., location information related to the route, captured images, etc.) without any details regarding the particular vehicle and / or driver / owner. For example, when uploading data according to a“high” privacy setting, the system 100 can not include the vehicle identification number (VIN) or the name of the vehicle driver or owner, and can instead transmit data such as captured images and / or limited location information related to the route.

[0094] Other privacy levels are contemplated. For example, the system 100 can transmit data to the server according to a“medium” privacy level and include additional information not included under the“high” privacy level, such as the make and / or model of the vehicle and / or the vehicle type (e.g., passenger car, sport utility vehicle, truck, etc.). In some embodiments, the system 100 can upload data according to a“low” privacy level. Under the“low” privacy level setting, the system 100 can upload data and include information sufficient to uniquely identify the particular vehicle, owner / driver, and / or a portion or all of the route traveled by the vehicle. Such“low” privacy level data can include, for example, one or more of the following: VIN, driver / owner name, vehicle’s starting point before departure, vehicle’s intended destination, vehicle’s make and / or model, vehicle’s type, etc.

[0095] Figure 2A is a diagrammatic side view representation of an exemplary vehicle imaging system consistent with the disclosed embodiments. Figure 2B is Figure 2A is a diagrammatic top view illustration of the embodiment shown in Figure 2BAs illustrated, the disclosed embodiments can include a vehicle 200 that includes the system 100 in its body, with the first image capture device 122 positioned near a rearview mirror of the vehicle 200 and / or near a driver of the vehicle 200, the second image capture device 124 positioned on or in a bumper region (e.g., one of the bumper regions 210) of the vehicle 200, and the processing unit 110.

[0096] As Figure 2C illustrated, both image capture devices 122 and 124 can be positioned near a rearview mirror of the vehicle 200 and / or near a driver of the vehicle 200. Additionally, while two image capture devices 122 and 124 are shown in Figure 2B and Figure 2C , it should be understood that other embodiments can include more than two image capture devices. For example, in the embodiments shown in Figure 2D and Figure 2E , a first image capture device 122, a second image capture device 124, and a third image capture device 126 are included in the system 100 of the vehicle 200.

[0097] As Figure 2D illustrated, the image capture device 122 can be positioned near a rearview mirror of the vehicle 200 and / or near a driver of the vehicle 200, while the image capture devices 124 and 126 can be positioned on or in a bumper region (e.g., one of the bumper regions 210) of the vehicle 200. And as Figure 2E illustrated, the image capture devices 122, 124, and 126 can be positioned near a rearview mirror of the vehicle 200 and / or near a driver seat of the vehicle 200. The disclosed embodiments are not limited to any particular number and configuration of image capture devices, and the image capture devices can be positioned anywhere within and / or on the vehicle 200 as appropriate.

[0098] It should be understood that the disclosed embodiments are not limited to vehicles, and can be applied to other contexts. It should also be understood that the disclosed embodiments are not limited to a particular type of vehicle 200, and can be applicable to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.

[0099] The first image capture device 122 can include any suitable type of image capture device. The image capture device 122 can include an optical axis. In one example, the image capture device 122 can include an Aptina M9V024 WVGA sensor with a global shutter. In other embodiments, the image capture device 122 can provide a resolution of 1280 x 960 pixels and can include a rolling shutter. The image capture device 122 can include various optical elements. In some embodiments, one or more lenses can be included, for example, to provide a desired focal length and field of view for the image capture device. In some embodiments, the image capture device 122 can be associated with a 6 mm lens or a 12 mm lens. In some embodiments, the image capture device 122 can be configured to capture images having a desired field of view (FOV) 202, as shown in Figure 2D For example, the image capture device 122 can be configured to have a regular FOV, such as in the range of 40 degrees to 56 degrees, including a 46 degree FOV, a 50 degree FOV, a 52 degree FOV, or a larger FOV. Alternatively, the image capture device 122 can be configured to have a narrow FOV in the range of 23 to 40 degrees, such as a 28 degree FOV or a 36 degree FOV. Further, the image capture device 122 can be configured to have a wide FOV in the range of 100 to 180 degrees. In some embodiments, the image capture device 122 can include a wide-angle bumper camera or a camera with up to a 180 degree FOV. In some embodiments, the image capture device 122 can be a 7.2M pixel image capture device having an aspect ratio of about 2: 1 (e.g., HxV = 3800 x 1900 pixels) and having a horizontal FOV of about 100 degrees. Such an image capture device can be used in place of a three image capture device configuration. Due to significant lens distortion, the vertical FOV of such an image capture device can be significantly less than 50 degrees in implementations in which the image capture device uses a radially symmetric lens. For example, such a lens can not be radially symmetric, which would allow for a vertical FOV greater than 50 degrees with a 100 degree horizontal FOV.

[0100] The first image capture device 122 can capture a plurality of first images relative to a scene associated with the vehicle 200. Each of the plurality of first images can be captured as a series of image scan lines, which can be captured using a rolling shutter. Each scan line can include a plurality of pixels.

[0101] The first image capture device 122 can have a scan rate associated with the capture of each image scan line of the first series of image scan lines. The scan rate can refer to a rate at which the image sensor is able to capture image data associated with each pixel included in a particular scan line.

[0102] The image capture devices 122, 124, and 126 can include any suitable type and number of image sensors, such as including a CCD sensor or a CMOS sensor. In one embodiment, a CMOS image sensor can be employed with a rolling shutter, such that each pixel in a row is read one at a time, and the scanning of the rows is performed on a row-by-row basis until the entire image frame has been captured. In some embodiments, the rows can be captured sequentially from top to bottom with respect to the frame.

[0103] In some embodiments, one or more of the image capture devices disclosed herein (e.g., the image capture devices 122, 124, and 126) can constitute a high resolution imager, and can have a resolution greater than 5M pixels, 7M pixels, 10M pixels, or more.

[0104] The use of a rolling shutter can result in pixels in different rows being exposed and captured at different times, which can result in skew and other image artifacts in the captured image frame. On the other hand, when the image capture device 122 is configured to operate with a global or synchronous shutter, all of the pixels can be exposed for the same amount of time and exposed within a common exposure period. As a result, the image data in a frame collected from a system employing a global shutter represents a snapshot of the entire FOV (such as the FOV 202) at a particular time. In contrast, in a rolling shutter application, each row in the frame is exposed at a different time, and the data is captured at different times. Thus, in an image capture device with a rolling shutter, a moving object can appear distorted. This phenomenon will be described in more detail below.

[0105] The second image capture device 124 and the third image capture device 126 can be any type of image capture device. Similar to the first image capture device 122, each of the image capture devices 124 and 126 can include an optical axis. In one embodiment, each of the image capture devices 124 and 126 can include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of the image capture devices 124 and 126 can include a rolling shutter. Similar to the image capture device 122, the image capture devices 124 and 126 can be configured to include various lenses and optical elements. In some embodiments, the lenses associated with the image capture devices 124 and 126 can provide a FOV (such as the FOVs 204 and 206) that is the same as or narrower than the FOV (such as the FOV 202) associated with the image capture device 122. For example, the image capture devices 124 and 126 can have a FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less.

[0106] The image capture devices 124 and 126 can acquire a plurality of second and third images of a scene associated with the vehicle 200. Each of the plurality of second and third images can be captured as a second series of image scan lines and a third series of image scan lines, which can be captured using a rolling shutter. Each scan line or scan row can have a plurality of pixels. The image capture devices 124 and 126 can have second and third scan rates associated with acquiring each of the image scan lines included in the second and third series.

[0107] Each of the image capture devices 122, 124, and 126 can be positioned at any suitable location and orientation relative to the vehicle 200. The relative positioning of the image capture devices 122, 124, and 126 can be selected to assist in fusing information acquired from the image capture devices together. For example, in some embodiments, a FOV associated with the image capture device 124, such as the FOV 204, can partially or completely overlap with a FOV associated with the image capture device 122, such as the FOV 202, and a FOV associated with the image capture device 126, such as the FOV 206.

[0108] The image capture devices 122, 124, and 126 can be located at any suitable relative height on the vehicle 200. In one instance, there can be a difference in height between the image capture devices 122, 124, and 126, which can provide sufficient parallax information to enable stereo analysis. For example, as shown in FIG. 2, the two image capture devices 122 and 124 are at different heights. There can also be a lateral displacement difference between the image capture devices 122, 124, and 126, giving additional parallax information for stereo analysis by the processing unit 110, for example. The difference in lateral displacement can be represented as d Figure 2A x Figures 2C-2D In some embodiments, there can be a fore-aft displacement (e.g., range displacement) between the image capture devices 122, 124, and 126. For example, the image capture device 122 can be located 0.5 to 2 meters or more behind the image capture device 124 and / or the image capture device 126. This type of displacement can enable one of the image capture devices to cover potential blind spots of the other image capture device(s).

[0109] ​​Image capture device 122 can have any suitable resolution capability (e.g., number of pixels associated with an image sensor), and the resolution of the image sensor(s) associated with image capture device 122 can be higher, lower, or the same as the resolution of the image sensor(s) associated with image capture devices 124 and 126. In some embodiments, the image sensor(s) associated with image capture device 122 and / or image capture devices 124 and 126 can have a resolution of 640x480, 1024x768, 1280x960, or any other suitable resolution.

[0110] The frame rate (e.g., the rate at which an image capture device acquires a set of pixel data for one image frame before proceeding to capture pixel data associated with a next image frame) can be controllable. The frame rate associated with image capture device 122 can be higher, lower, or the same as the frame rate associated with image capture devices 124 and 126. The frame rate associated with image capture devices 122, 124, and 126 can depend on various factors that can affect timing of the frame rate. For example, one or more of image capture devices 122, 124, and 126 can include an optional pixel delay period that is applied before or after acquisition of image data associated with one or more pixels of an image sensor in image capture device 122, 124, and / or 126. Generally, image data corresponding to each pixel can be acquired according to a clock rate of the device (e.g., one pixel per clock cycle). Additionally, in embodiments that include a rolling shutter, one or more of image capture devices 122, 124, and 126 can include an optional horizontal blanking period that is applied before or after acquisition of image data associated with a row of pixels of an image sensor in image capture device 122, 124, and / or 126. Furthermore, one or more of image capture devices 122, 124, and / or 126 can include an optional vertical blanking period that is applied before or after acquisition of image data associated with an image frame of image capture device 122, 124, and 126.

[0111] These timing controls can enable synchronization of frame rates associated with image capture devices 122, 124, and 126, even in situations where the line scan rate of each device is different. Additionally, as will be discussed in greater detail below, these optional timing controls can enable synchronization of image capture from areas in which the FOV of image capture device 122 overlaps with one or more FOVs of image capture devices 124 and 126, even in situations where the field of view of image capture device 122 is different from the FOVs of image capture devices 124 and 126, among other factors (e.g., image sensor resolution, maximum line scan rate, etc.).

[0112] The frame rate timing in image capture devices 122, 124, and 126 can depend on the resolution of the associated image sensor. For example, assuming similar line scan rates for the two devices, if one device includes an image sensor having a resolution of 640x480 and the other device includes an image sensor having a resolution of 1280x960, more time will be required to acquire a frame of image data from the sensor with the higher resolution.

[0113] Another factor that can affect the timing of image data acquisition in image capture devices 122, 124, and 126 is the maximum line scan rate. For example, acquiring a line of image data from an image sensor included in image capture devices 122, 124, and 126 will require some minimum amount of time. Assuming no pixel delay period is added, this minimum amount of time for acquiring a line of image data will be related to the maximum line scan rate of the particular device. A device that provides a higher maximum line scan rate has the potential to provide a higher frame rate than a device with a lower maximum line scan rate. In some embodiments, one or more of image capture devices 124 and 126 can have a maximum line scan rate that is higher than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture device 124 and / or 126 can be 1.25, 1.5, 1.75, or 2 times or more than the maximum line scan rate of image capture device 122.

[0114] In another embodiment, image capture devices 122, 124, and 126 can have the same maximum line scan rate, but image capture device 122 can operate at a scan rate that is less than or equal to its maximum scan rate. The system can be configured such that one or more of image capture devices 124 and 126 operate at a line scan rate that is equal to the line scan rate of image capture device 122. In other instances, the system can be configured such that the line scan rate of image capture device 124 and / or image capture device 126 can be 1.25, 1.5, 1.75, or 2 times or more than the line scan rate of image capture device 122.

[0115] In some embodiments, the image capture devices 122, 124, and 126 can be asymmetric. That is, they can include cameras having different fields of view (FOV) and focal lengths. For example, the fields of view of the image capture devices 122, 124, and 126 can include any desired area relative to the environment of the vehicle 200. In some embodiments, one or more of the image capture devices 122, 124, and 126 can be configured to capture image data from the environment in front of the vehicle 200, behind the vehicle 200, to the side of the vehicle 200, or a combination thereof.

[0116] Further, the focal length associated with each image capture device 122, 124, and / or 126 can be selectable (e.g., by including appropriate lenses, etc.) such that each device captures images of objects within a desired range of distances relative to the vehicle 200. For example, in some embodiments, the image capture devices 122, 124, and 126 can capture images of close-up objects within a few meters of the vehicle. The image capture devices 122, 124, and 126 can also be configured to capture images of objects within a range farther from the vehicle (e.g., 25 m, 50 m, 100 m, 150 m, or more). Further, the focal lengths of the image capture devices 122, 124, and 126 can be selected such that one image capture device (e.g., the image capture device 122) can capture images of objects relatively close to the vehicle (e.g., within 10 m or within 20 m), while other image capture devices (e.g., the image capture devices 124 and 126) can capture images of objects farther from the vehicle 200 (e.g., more than 20 m, 50 m, 100 m, 150 m, etc., from the vehicle).

[0117] According to some embodiments, the FOV of one or more of the image capture devices 122, 124, and 126 can have a wide angle. For example, it can be advantageous for the image capture devices 122, 124, and 126 that can be used to capture images of the area proximate to the vehicle 200 to have a FOV of 140 degrees, for example. For example, the image capture device 122 can be used to capture images of the area to the right or left of the vehicle 200, and in such embodiments, it can be desirable for the image capture device 122 to have a wide FOV (e.g., at least 140 degrees).

[0118] The field of view associated with each of the image capture devices 122, 124, and 126 can depend on the respective focal length. For example, as the focal length increases, the corresponding field of view decreases.

[0119] Image capture devices 122, 124, and 126 can be configured to have any suitable fields of view. In one particular example, image capture device 122 can have a horizontal FOV of 46 degrees, image capture device 124 can have a horizontal FOV of 23 degrees, and image capture device 126 can have a horizontal FOV between 23 degrees and 46 degrees. In another example, image capture device 122 can have a horizontal FOV of 52 degrees, image capture device 124 can have a horizontal FOV of 26 degrees, and image capture device 126 can have a horizontal FOV between 26 degrees and 52 degrees. In some embodiments, the ratio of the FOV of image capture device 122 to the FOV of image capture device 124 and / or image capture device 126 can vary from 1.5 to 2.0. In other embodiments, the ratio can vary between 1.25 and 2.25.

[0120] System 100 can be configured such that the field of view of image capture device 122 at least partially or completely overlaps with the field of view of image capture device 124 and / or image capture device 126. In some embodiments, system 100 can be configured such that the fields of view of image capture devices 124 and 126 fall, for example, within (e.g., are narrower than) the field of view of image capture device 122 and share a common center with the field of view of image capture device 122. In other embodiments, image capture devices 122, 124, and 126 can capture adjacent FOVs, or can have partial overlap in their FOVs. In some embodiments, the fields of view of image capture devices 122, 124, and 126 can be aligned such that the center of image capture devices 124 and / or 126, which have narrower FOVs, can be located in the lower half of the field of view of image capture device 122, which has a wider FOV.

[0121] Figure 2F is a diagrammatic representation of an exemplary vehicle control system consistent with the disclosed embodiments. As Figure 2FAs indicated, vehicle 200 may include a throttle system 220, a braking system 230, and a steering system 240. System 100 may provide inputs (e.g., control signals) to one or more of the throttle system 220, braking system 230, and steering system 240 via one or more data links (e.g., any one or more wired and / or wireless links for transmitting data). For example, based on analysis of images acquired by image capture devices 122, 124, and / or 126, system 100 may provide control signals to one or more of the throttle system 220, braking system 230, and steering system 240 to navigate vehicle 200 (e.g., by inducing acceleration, turning, lane departure, etc.). Further, system 100 may receive inputs from one or more of the throttle system 220, braking system 230, and steering system 240 indicating the operating status of vehicle 200 (e.g., speed, whether vehicle 200 is braking or turning, etc.). The following is in conjunction with... Figures 4-7 Further details will be provided.

[0122] like Figure 3A As shown, vehicle 200 may also include a user interface 170 for interacting with the driver or passengers of vehicle 200. For example, user interface 170 in a vehicle application may include a touchscreen 320, a knob 330, a button 340, and a microphone 350. The driver or passengers of vehicle 200 may also interact with system 100 using handles (e.g., handles located on or near the steering column of vehicle 200, including, for example, turn signal handles), buttons (e.g., buttons located on the steering wheel of vehicle 200), etc. In some embodiments, microphone 350 may be positioned adjacent to rearview mirror 310. Similarly, in some embodiments, image capture device 122 may be located near rearview mirror 310. In some embodiments, user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, system 100 may provide various notifications (e.g., alarms) via speakers 360.

[0123] Figures 3B-3D This is an illustration of an exemplary camera mount 370 conforming to the disclosed embodiments, which is configured to be located behind a rearview mirror (e.g., rearview mirror 310) and abut against the windshield of a vehicle. Figure 3BAs shown, camera mount 370 can include image capture devices 122, 124, and 126. Image capture devices 124 and 126 can be positioned behind a glare shield 380, which can be flush against a vehicle windshield and include a combination of film and / or anti-reflective material. For example, glare shield 380 can be positioned such that the shield is aligned against a vehicle windshield having a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 can be positioned behind glare shield 380, for example, as depicted in FIG. 12. The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124, and 126, camera mount 370, and glare shield 380. Figure 3D As shown, camera mount 370 can include image capture devices 122, 124, and 126. Image capture devices 124 and 126 can be positioned behind a glare shield 380, which can be flush against a vehicle windshield and include a combination of film and / or anti-reflective material. For example, glare shield 380 can be positioned such that the shield is aligned against a vehicle windshield having a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 can be positioned behind glare shield 380, for example, as depicted in FIG. 12. The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124, and 126, camera mount 370, and glare shield 380. Figure 3C As shown, camera mount 370 can include image capture devices 122, 124, and 126. Image capture devices 124 and 126 can be positioned behind a glare shield 380, which can be flush against a vehicle windshield and include a combination of film and / or anti-reflective material. For example, glare shield 380 can be positioned such that the shield is aligned against a vehicle windshield having a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 can be positioned behind glare shield 380, for example, as depicted in FIG. 12. The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124, and 126, camera mount 370, and glare shield 380. Figure 3B As shown, camera mount 370 can include image capture devices 122, 124, and 126. Image capture devices 124 and 126 can be positioned behind a glare shield 380, which can be flush against a vehicle windshield and include a combination of film and / or anti-reflective material. For example, glare shield 380 can be positioned such that the shield is aligned against a vehicle windshield having a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 can be positioned behind glare shield 380, for example, as depicted in FIG. 12. The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124, and 126, camera mount 370, and glare shield 380.

[0124] As those skilled in the art will appreciate in light of the disclosures herein, many modifications and / or changes can be made to the aforementioned disclosed embodiments. For example, not all components can be necessary for operation of system 100. Further, any component can be located in any suitable portion of system 100, and components can be rearranged into various configurations while providing the functionality of the disclosed embodiments. Accordingly, the foregoing configurations are examples, and system 100 can provide a wide range of functionality to analyze the surroundings of vehicle 200 and navigate vehicle 200 in response to the analysis, regardless of the configurations discussed above.

[0125] As discussed in further detail below and consistent with various disclosed embodiments, system 100 can provide various features related to autonomous driving and / or driver assist technology. For example, system 100 can analyze image data, location data (e.g., GPS location information), map data, speed data, and / or data from sensors included in vehicle 200. System 100 can collect data from, for example, image acquisition unit 120, location sensor 130, and other sensors for analysis. Further, system 100 can analyze the collected data to determine whether vehicle 200 should take some action, and subsequently take the determined action automatically without human intervention. For example, when vehicle 200 is navigating without human intervention, system 100 can automatically control the braking, acceleration, and / or steering of vehicle 200 (e.g., by sending control signals to one or more of throttle system 220, brake system 230, and steering system 240). Further, system 100 can analyze the collected data and issue warnings and / or alerts to vehicle occupants based on the analysis of the collected data. Additional details regarding embodiments provided by system 100 are provided below.

[0126] Forward Multi-Imaging System

[0127] As discussed above, the system 100 can provide driving assistance functionality using a multi-camera system. The multi-camera system can use one or more cameras facing in the direction of travel of the vehicle. In other embodiments, the multi-camera system can include one or more cameras facing to the side of the vehicle or to the rear of the vehicle. In one embodiment, for example, the system 100 can use a dual camera imaging system in which a first camera and a second camera (e.g., image capture devices 122 and 124) can be positioned at the front and / or sides of the vehicle (e.g., vehicle 200). The field of view of the first camera can be greater than, less than, or partially overlapping with the field of view of the second camera. Further, the first camera can be connected to a first image processor for performing monocular image analysis on images provided by the first camera, and the second camera can be connected to a second image processor for performing monocular image analysis on images provided by the second camera. The outputs (e.g., processed information) of the first and second image processors can be combined. In some embodiments, the second image processor can receive images from both the first camera and the second camera to perform stereo analysis. In another embodiment, the system 100 can use a three camera imaging system in which each of the cameras has a different field of view. Accordingly, such a system can make decisions based on information derived from objects located at different distances in front of and to the sides of the vehicle. References to monocular image analysis can refer to instances in which image analysis is performed based on images captured from a single viewpoint (e.g., from a single camera). Stereo image analysis can refer to instances in which image analysis is performed based on two or more images captured with one or more variations of image capture parameters. For example, images captured suitable for performing stereo image analysis can include images captured from two or more different positions, images captured from different fields of view, images captured using different focal lengths, images captured with parallax information, etc.

[0128] For example, in one embodiment, system 100 can implement a three-camera configuration using image capture devices 122, 124, and 126. In such a configuration, image capture device 122 can provide a narrow field of view (e.g., 34 degrees or other values selected from a range of approximately 20 to 45 degrees, etc.), image capture device 124 can provide a wide field of view (e.g., 150 degrees or other values selected from a range of approximately 100 to approximately 180 degrees), and image capture device 126 can provide an intermediate field of view (e.g., 46 degrees or other values selected from a range of approximately 35 degrees to approximately 60 degrees). In some embodiments, image capture device 126 can act as a main or primary camera. Image capture devices 122, 124, and 126 can be positioned behind rearview mirror 310 and positioned substantially side-by-side (e.g., 6 cm apart). Further, in some embodiments, as discussed above, one or more of image capture devices 122, 124, and 126 can be mounted behind a glare shield 380 that is flush with the windshield of vehicle 200. Such a shield can be used to minimize the effects of any reflections from inside the vehicle on image capture devices 122, 124, and 126.

[0129] In another embodiment, as discussed above in connection with Figure 3B and Figure 3C wide field of view camera (e.g., image capture device 124 in the example above) can be mounted lower than the narrow and main field of view cameras (e.g., image devices 122 and 126 in the example above). This configuration can provide a clear line of sight from the wide field of view camera. To reduce reflections, the camera can be mounted close to the windshield of vehicle 200 and can include a polarizer on the camera to suppress reflected light.

[0130] A three-camera system can provide certain performance characteristics. For example, some embodiments can include the ability to verify the detection of an object by one camera based on the detection results from another camera. In the three-camera configuration discussed above, processing unit 110 can include, for example, three processing devices (e.g., three EyeQ series processor chips as discussed above), with each processing device dedicated to processing images captured by one or more of image capture devices 122, 124, and 126.

[0131] In a three camera system, the first processing device can receive images from both the primary camera and the narrow field of view camera and perform vision processing on the narrow FOV camera to, for example, detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Further, the first processing device can compute the disparity of pixels between images from the primary camera and images from the narrow camera and create a 3D reconstruction of the environment of the vehicle 200. The first processing device can then combine the 3D reconstruction with 3D map data or with 3D information computed based on information from another camera.

[0132] The second processing device can receive images from the primary camera and perform vision processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Additionally, the second processing device can compute camera displacement and, based on that displacement, compute the disparity of pixels between successive images and create a 3D reconstruction of the scene (e.g., structure from motion). The second processing device can send the 3D reconstruction based on structure from motion to the first processing device to combine with the stereo 3D images.

[0133] The third processing device can receive images from the wide FOV camera and process those images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device can further execute additional processing instructions to analyze the images to identify objects in the images that are moving, such as vehicles changing lanes, pedestrians, etc.

[0134] In some embodiments, having information based on image streams captured and processed independently can provide opportunities for providing redundancy in the system. Such redundancy can include, for example, using a first image capture device and processed images from that device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.

[0135] In some embodiments, system 100 may use two image capture devices (e.g., image capture devices 122 and 124) to provide navigation assistance to vehicle 200 and a third image capture device (e.g., image capture device 126) to provide redundancy and verify the analysis of data received from the other two image capture devices. For example, in such a configuration, image capture devices 122 and 124 may provide images for navigation of vehicle 200 by stereo analysis performed by system 100, while image capture device 126 may provide images for monocular analysis performed by system 100 to provide redundancy and verification of information based on images captured from image capture devices 122 and / or 124. That is, image capture device 126 (and corresponding processing devices) may be considered to provide a redundant subsystem for checking the analysis derived from image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AEB) system). Furthermore, in some embodiments, the redundancy and verification of received information can be supplemented based on information received from one or more sensors (e.g., radar, lidar, acoustic sensors, information received from one or more transceivers outside the vehicle, etc.).

[0136] Those skilled in the art will recognize that the camera configurations, camera placements, number of cameras, camera positions, etc., described above are merely examples. These components, and other components described relative to the overall system, can be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of a multi-camera system to provide driver assistance and / or autonomous vehicle functionality are provided below.

[0137] Figure 4 This is an exemplary functional block diagram of a memory 140 and / or 150 that can store / program instructions for performing one or more operations conforming to the disclosed embodiments. Although memory 140 is referred to below, those skilled in the art will recognize that instructions can be stored in memory 140 and / or 150.

[0138] like Figure 4 As shown, memory 140 may store monocular image analysis module 402, stereo image analysis module 404, velocity and acceleration module 406, and navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Further, application processor 180 and / or image processor 190 may execute instructions stored in any of the modules 402, 404, 406, and 408 included in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may individually or collectively refer to application processor 180 and image processor 190. Accordingly, steps of any of the processes in the following procedures may be performed by one or more processing devices.

[0139] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) for performing monocular image analysis on a set of images acquired by one of the image capture devices 122, 124, and 126 when executed by the processing unit 110. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensing information (e.g., information from radar, lidar, etc.) to perform monocular image analysis. (The following is in conjunction with...) Figures 5A-5D As described, the monocular image analysis module 402 may include instructions for detecting a set of features in the image set, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other features associated with the vehicle's environment. Based on this analysis, system 100 (e.g., via processing unit 110) may induce one or more navigation responses in vehicle 200, such as turning, lane departure, changes in acceleration, etc., as discussed below in conjunction with navigation response module 408.

[0140] In one embodiment, the stereo image analysis module 404 may store instructions (such as computer vision software) for performing stereo image analysis on a first set and a second set of images acquired by a combination of image capture devices selected from image capture devices 122, 124, and 126 when executed by the processing unit 110. In some embodiments, the processing unit 110 may combine information from the first set and the second set of images with additional sensing information (e.g., information from radar) to perform stereo image analysis. For example, the stereo image analysis module 404 may include instructions for performing stereo image analysis based on a first set of images acquired by image capture device 124 and a second set of images acquired by image capture device 126. (The following is a continuation of the previous paragraph.) Figure 6 As described, the stereo image analysis module 404 may include instructions for detecting sets of features within a first set and a second set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, etc. Based on the analysis, the processing unit 110 may induce one or more navigation responses in the vehicle 200, such as turning, lane departure, changes in acceleration, etc., as discussed below in conjunction with the navigation response module 408. Furthermore, in some embodiments, the stereo image analysis module 404 may implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems, such as systems configured to use computer vision algorithms to detect and / or label objects in an environment that captures and processes sensor information from them. In one embodiment, the stereo image analysis module 404 and / or other image processing modules may be configured to use a combination of trained and untrained systems.

[0141] In one embodiment, the speed and acceleration module 406 can store software configured to analyze data received from one or more computing and electromechanical devices in the vehicle 200 configured to cause changes in the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 can execute instructions associated with the speed and acceleration module 406 to calculate a target speed of the vehicle 200 based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data can include, for example, a target location, speed and / or acceleration, a position and / or speed of the vehicle 200 relative to nearby vehicles, pedestrians, or road objects, position information of the vehicle 200 relative to lane markers of the road, etc. In addition, the processing unit 110 can calculate a target speed of the vehicle 200 based on sensory inputs (e.g., information from radar) and inputs from other systems of the vehicle 200, such as the throttling system 220, the braking system 230, and / or the steering system 240 of the vehicle 200. Based on the calculated target speed, the processing unit 110 can transmit electronic signals to the throttling system 220, the braking system 230, and / or the steering system 240 of the vehicle 200 to trigger changes in speed and / or acceleration by, for example, physically depressing the brakes or releasing the accelerator of the vehicle 200.

[0142] In one embodiment, the navigation response module 408 can store software executable by the processing unit 110 to determine a desired navigation response based on data derived from execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data can include position and velocity information associated with nearby vehicles, pedestrians, and road objects, target position information for the vehicle 200, etc. Additionally, in some embodiments, the navigation response can be based (partially or completely) on map data, a predetermined position of the vehicle 200, and / or a relative velocity or relative acceleration between the vehicle 200 and one or more objects detected from execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 can also determine a desired navigation response based on sensory input (e.g., information from radar) and input from other systems of the vehicle 200, such as the throttling system 220, the braking system 230, and the steering system 240 of the vehicle 200. Based on the desired navigation response, the processing unit 110 can transmit electronic signals to the throttling system 220, the braking system 230, and the steering system 240 of the vehicle 200 to trigger the desired navigation response by, for example, turning the steering wheel of the vehicle 200 to achieve a predetermined angle of rotation. In some embodiments, the processing unit 110 can use the output of the navigation response module 408 (e.g., the desired navigation response) as an input to the execution velocity and acceleration module 406 for calculating a change in velocity of the vehicle 200.

[0143] Additionally, any of the modules disclosed herein (e.g., modules 402, 404, and 406) can implement techniques associated with trained systems, such as neural networks or deep neural networks, or untrained systems.

[0144] Figure 5A FIG. 5A is a flowchart illustrating an exemplary process 500A for causing one or more navigation responses based on monocular images, consistent with disclosed embodiments. At step 510, the processing unit 110 can receive a plurality of images via the data interface 128 between the processing unit 110 and the image capture unit 120. For example, a camera included in the image capture unit 120, such as the image capture device 122 having a field of view 202, can capture a plurality of images of an area in front of the vehicle 200 (or, for example, to the side or rear of the vehicle), and transmit the images to the processing unit 110 over a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). At step 520, the processing unit 110 can execute the monocular image analysis module 402 to analyze the plurality of images, as described below in connection with FIG. 5B. Figures 5B-5DFurther details are described in further detail. By performing the analysis, processing unit 110 can detect a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and the like.

[0145] At step 520, processing unit 110 can also execute monocular image analysis module 402 to detect various road hazards, such as, for example, a piece of a truck tire, a fallen road sign, loose cargo, a small animal, and the like. Road hazards can vary in structure, shape, size, and color, which can make detecting such hazards more challenging. In some embodiments, processing unit 110 can execute monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, processing unit 110 can estimate camera motion between consecutive image frames and compute the disparity of pixels between the frames to construct a 3D map of the road. Processing unit 110 can then use the 3D map to detect the road surface and hazards present on the road surface.

[0146] At step 530, processing unit 110 can execute navigation response module 408 to cause one or more navigational responses based on the analysis performed at step 520 and as described above in connection with FIG. 4. Figure 4 The described techniques cause one or more navigational responses in vehicle 200. The navigational responses can include, for example, a turn, a lane shift, a change in acceleration, and the like. In some embodiments, processing unit 110 can use data derived from the execution of velocity and acceleration module 406 to cause one or more navigational responses. Further, multiple navigational responses can occur simultaneously, in sequence, or in any combination of these ways. For example, processing unit 110 can cause vehicle 200 to shift through one lane and then accelerate by, for example, sequentially transmitting control signals to steering system 240 and throttle system 220 of vehicle 200. Alternatively, processing unit 110 can cause vehicle 200 to brake while shifting lanes by, for example, simultaneously transmitting control signals to braking system 230 and steering system 240 of vehicle 200.

[0147] Figure 5Bis a flowchart illustrating an exemplary process 500B for detecting one or more vehicles and / or pedestrians in a set of images consistent with the disclosed embodiments. Processing unit 110 can execute monocular image analysis module 402 to implement process 500B. At step 540, processing unit 110 can determine a set of candidate objects representing possible vehicles and / or pedestrians. For example, processing unit 110 can scan one or more images, compare the images to one or more predetermined patterns, and identify possible locations within each image that can contain objects of interest (e.g., vehicles, pedestrians, or portions thereof). The predetermined patterns can be designed in a manner that achieves a high "false hit rate" and a low "miss hit rate." For example, processing unit 110 can use a low similarity threshold for the predetermined patterns to identify candidate objects as possible vehicles or pedestrians. Doing so can allow processing unit 110 to reduce the probability of missing (e.g., failing to identify) a candidate object representing a vehicle or pedestrian.

[0148] At step 542, processing unit 110 can filter the set of candidate objects to exclude certain candidates (e.g., irrelevant or less relevant objects) based on classification criteria. Such criteria can be derived from various attributes associated with object types stored in a database (e.g., a database stored in memory 140). Attributes can include object shape, size, texture, location (e.g., relative to the location of vehicle 200), and so forth. Thus, processing unit 110 can use one or more sets of criteria to reject false candidates from the set of candidate objects.

[0149] At step 544, processing unit 110 can analyze multiple frames of the images to determine whether objects in the set of candidate objects represent vehicles and / or pedestrians. For example, processing unit 110 can track detected candidate objects across consecutive frames and accumulate frame-by-frame data (e.g., size, location relative to vehicle 200, etc.) associated with the detected objects. Additionally, processing unit 110 can estimate parameters of the detected objects and compare frame-by-frame location data of the objects to predicted locations.

[0150] At step 546, processing unit 110 can construct a set of measurements for the detected objects. Such measurements can include, for example, position, velocity, and acceleration values associated with the detected objects (relative to vehicle 200). In some embodiments, processing unit 110 can construct the measurements based on estimation techniques that use a series of time-based observations, such as a Kalman filter or linear quadratic estimation (LQE), and / or based on available modeling data for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). The Kalman filter can be based on a measurement of object scale, where the scale measurement is proportional to the time of impact (e.g., the amount of time vehicle 200 takes to reach the object). Thus, by performing steps 540-546, processing unit 110 can identify vehicles and pedestrians present in the set of captured images and derive information associated with the vehicles and pedestrians (e.g., position, velocity, size). Based on the identified and derived information, processing unit 110 can cause one or more navigational responses in vehicle 200, as described above in connection with FIG. 5A. Figure 5A

[0151] At step 548, processing unit 110 can perform an optical flow analysis of one or more images to reduce the probability of detecting "false hits" and missed candidates representing vehicles or pedestrians. The optical flow analysis can refer to, for example, analyzing motion patterns in one or more images associated with other vehicles and pedestrians that are different from the motion of the road surface relative to vehicle 200. Processing unit 110 can calculate the motion of a candidate object by observing different positions of the object across multiple image frames captured at different times. Processing unit 110 can use the position and time values as inputs to a mathematical model to calculate the motion of the candidate object. Thus, the optical flow analysis can provide another method of detecting vehicles and pedestrians in the vicinity of vehicle 200. Processing unit 110 can perform the optical flow analysis in conjunction with steps 540-546, thereby providing redundancy for detecting vehicles and pedestrians and increasing the reliability of system 100.

[0152] Figure 5C ​is a flowchart illustrating an exemplary process 500C for detecting road markings and / or lane geometry information in a set of images, consistent with the disclosed embodiments. The processing unit 110 can execute the monocular image analysis module 402 to implement the process 500C. At step 550, the processing unit 110 can detect a set of objects by scanning one or more images. To detect segments with lane markings, lane geometry information, and other pertinent road markings, the processing unit 110 can filter the set of objects to exclude those objects that are determined to be irrelevant (e.g., small potholes, small rocks, etc.). At step 552, the processing unit 110 can group together segments belonging to the same road marking or lane marking that are detected at step 550. Based on the grouping, the processing unit 110 can form a model, such as a mathematical model, to represent the detected segments.

[0153] At step 554, the processing unit 110 can construct a set of measurements associated with the detected segments. In some embodiments, the processing unit 110 can create a projection of the detected segments from the image plane to the real-world plane. The projection can be characterized using a cubic polynomial with coefficients corresponding to physical properties, such as the location, slope, curvature, and curvature derivative of the detected road. In generating the projection, the processing unit 110 can take into account changes in the road surface as well as the pitch and roll rates associated with the vehicle 200. Further, the processing unit 110 can model the road elevation by analyzing position and motion cues present on the road surface. Further still, the processing unit 110 can estimate the pitch and roll rates associated with the vehicle 200 by tracking a set of feature points in one or more images.

[0154] At step 556, the processing unit 110 can perform multi-frame analysis by, for example, tracking the detected segments across consecutive image frames and accumulating frame-by-frame data associated with the detected segments. As the processing unit 110 performs multi-frame analysis, the set of measurements constructed at step 554 can become more reliable and associated with an increasingly high level of confidence. Thus, by performing steps 550, 552, 554, and 556, the processing unit 110 can identify road markings present within the set of captured images and derive lane geometry information. Based on the identified and derived information, the processing unit 110 can cause one or more navigational responses in the vehicle 200, as described above in connection with FIG. 4. Figure 5A

[0155] ​At step 558, processing unit 110 can consider additional sources of information to further form a safety model for vehicle 200 in the context of the surroundings of vehicle 200. Processing unit 110 can use the safety model to define a context in which system 100 can perform autonomous control of vehicle 200 in a safe manner. To form the safety model, in some embodiments, processing unit 110 can consider the positions and motions of other vehicles, detected road edges and obstacles, and / or general road shape descriptions extracted from map data, such as data from map database 160. By considering additional sources of information, processing unit 110 can provide redundancy for detecting road markings and lane geometry, and improve the reliability of system 100.

[0156] Figure 5D FIG. 5D is a flowchart illustrating an exemplary process 500D for detecting traffic lights in a set of images, consistent with disclosed embodiments. Processing unit 110 can execute monocular image analysis module 402 to implement process 500D. At step 560, processing unit 110 can scan the set of images and identify objects appearing in the images at locations that are likely to contain traffic lights. For example, processing unit 110 can filter the identified objects to construct a set of candidate objects, excluding those that are unlikely to correspond to traffic lights. The filtering can be done based on various attributes associated with traffic lights, such as shape, size, texture, position (e.g., relative to vehicle 200), and the like. Such attributes can be based on multiple examples of traffic lights and traffic control signals, and stored in a database. In some embodiments, processing unit 110 can perform multi-frame analysis on the set of candidate objects reflecting possible traffic lights. For example, processing unit 110 can track the candidate objects across consecutive image frames, estimate the real-world positions of the candidate objects, and filter out those objects that are moving (they are unlikely to be traffic lights). In some embodiments, processing unit 110 can perform color analysis on the candidate objects, and identify the relative positions at which the detected colors appear inside possible traffic lights.

[0157] At step 562, processing unit 110 can analyze the geometry of the intersection. The analysis can be based on any combination of (i) the number of lanes detected on either side of vehicle 200, (ii) the markings detected on the road, such as arrow markings, and (iii) a description of the intersection extracted from map data, such as data in map database 160. Processing unit 110 can use information derived from the execution of monocular analysis module 402 to conduct the analysis. In addition, processing unit 110 can determine a correspondence between the traffic lights detected at step 560 and the lanes appearing in the vicinity of vehicle 200.

[0158] As the vehicle 200 gradually approaches the intersection, at step 564, the processing unit 110 can update the confidence level associated with the analyzed intersection geometry and the detected traffic lights. For example, a comparison of the estimated number of traffic lights present at the intersection with the actual number of traffic lights present at the intersection can influence the confidence level. Accordingly, based on the confidence level, the processing unit 110 can delegate control to the driver of the vehicle 200 to improve safety conditions. By performing steps 560, 562, and 564, the processing unit 110 can identify traffic lights present within the set of captured images and analyze intersection geometry information. Based on the identification and analysis, the processing unit 110 can cause one or more navigational responses in the vehicle 200, as described above in connection with FIG. 4, for example. Figure 5A

[0159] Figure 5E is a flowchart illustrating an exemplary process 500E for causing one or more navigational responses in a vehicle 200 based on a vehicle path, consistent with the disclosed embodiments. At step 570, the processing unit 110 can construct an initial vehicle path associated with the vehicle 200. The vehicle path can be represented using a set of points expressed in coordinates (x, z), and the distance d i between two points in the set of points can fall in the range of 1 meter to 5 meters. In one embodiment, the processing unit 110 can use two polynomials (e.g., a left road polynomial and a right road polynomial) to construct the initial vehicle path. The processing unit 110 can calculate the geometric midpoint between the two polynomials and offset each point included in the resulting vehicle path by a predetermined offset (e.g., a smart lane offset) if there is a predetermined offset (a zero offset can correspond to driving in the middle of the lane). The offset can be in a direction perpendicular to a segment between any two points in the vehicle path. In another embodiment, the processing unit 110 can use one polynomial and an estimated lane width to offset each point of the vehicle path by half of the estimated lane width plus a predetermined offset (e.g., a smart lane offset).

[0160] At step 572, the processing unit 110 can update the vehicle path constructed at step 570. The processing unit 110 can reconfigure the vehicle path constructed at step 570 using a higher resolution such that the distance d k between two points in the set of points representing the vehicle path is less than the distance d i described above. For example, the distance d k ​May fall in the range of 0.1 meters to 0.3 meters. The processing unit 110 can reconstruct the vehicle path using a parabolic spline algorithm, which can produce a cumulative distance vector S corresponding to the total length of the vehicle path (i.e., based on the set of points representing the vehicle path).

[0161] At step 574, the processing unit 110 can determine a lookahead point (expressed in coordinates as (x l , z l )) based on the updated vehicle path constructed at step 572. The processing unit 110 can extract the lookahead point from the cumulative distance vector S, and the lookahead point can be associated with a lookahead distance and a lookahead time. The lookahead distance, which can have a lower bound ranging from 10 meters to 20 meters, can be computed as the product of the velocity of the vehicle 200 and the lookahead time. For example, as the velocity of the vehicle 200 decreases, the lookahead distance can also decrease (e.g., until the lookahead distance reaches the lower bound). The lookahead time, which can range from 0.5 seconds to 1.5 seconds, can be inversely proportional to the gain of one or more control loops (such as a heading error tracking control loop) associated with eliciting a navigational response in the vehicle 200. For example, the gain of the heading error tracking control loop can depend on the bandwidth of a yaw rate loop, a steering actuator loop, vehicle lateral dynamics, etc. Thus, the higher the gain of the heading error tracking control loop, the shorter the lookahead time.

[0162] At step 576, the processing unit 110 can determine a heading error and a yaw rate command based on the lookahead point determined at step 574. The processing unit 110 can determine the heading error by computing the arctangent (e.g., arctan(x l / z l )) of the lookahead point. The processing unit 110 can determine the yaw rate command as the product of the heading error and a high-level control gain. If the lookahead distance is not at the lower bound, the high-level control gain can be equal to: (2 / lookahead time). Otherwise, the high-level control gain can be equal to: (2*velocity of the vehicle 200 / lookahead distance).

[0163] Figure 5F is a flowchart illustrating an example process 500F for determining whether a preceding vehicle is changing lanes, consistent with the disclosed embodiments. At step 580, the processing unit 110 can determine navigational information associated with a preceding vehicle (e.g., a vehicle traveling ahead of the vehicle 200). For example, the processing unit 110 can determine the position, velocity (e.g., direction and speed), and / or acceleration of the preceding vehicle using the techniques described above in connection with Figure 5A and Figure 5B At step 582, the processing unit 110 can determine whether the preceding vehicle is changing lanes based on the navigational information determined at step 580. For example, the processing unit 110 can determine whether the preceding vehicle is changing lanes using the techniques described above in connection with Figure 5EThe described techniques to determine one or more road polynomials, a lookahead point (associated with the vehicle 200), and / or a snail trail (e.g., a collection of points that describe a path taken by a preceding vehicle).

[0164] At step 582, the processing unit 110 can analyze the navigation information determined at step 580. In one embodiment, the processing unit 110 can calculate a distance between the snail trail and a road polynomial (e.g., along the trail). If the distance varies along the trail by more than a predetermined threshold (e.g., 0.1 to 0.2 meters on a straight road, 0.3 to 0.4 meters on a moderately curved road, and 0.5 to 0.6 meters on a road with sharp turns), the processing unit 110 can determine that the preceding vehicle can be changing lanes. In cases where multiple vehicles are detected to be driving ahead of the vehicle 200, the processing unit 110 can compare the snail trails associated with each vehicle. Based on the comparison, the processing unit 110 can determine that a vehicle whose snail trail does not match the snail trails of the other vehicles can be changing lanes. The processing unit 110 can additionally compare the curvature of the snail trail (associated with the preceding vehicle) to an expected curvature of the road segment in which the preceding vehicle is driving. The expected curvature can be extracted from map data (e.g., data from the map database 160), from road polynomials, from snail trails of other vehicles, from prior knowledge about the road, and so on. If the curvature of the snail trail differs from the expected curvature of the road segment by more than a predetermined threshold, the processing unit 110 can determine that the preceding vehicle can be changing lanes.

[0165] In another embodiment, the processing unit 110 can compare the instantaneous position of the preceding vehicle to the lookahead point (associated with the vehicle 200) over a particular temporal interval (e.g., 0.5 to 1.5 seconds). If the distance between the instantaneous position of the preceding vehicle and the lookahead point varies during the particular temporal interval, and the cumulative sum of the variations exceeds a predetermined threshold (e.g., 0.3 to 0.4 meters on a straight road, 0.7 to 0.8 meters on a moderately curved road, and 1.3 to 1.7 meters on a road with sharp turns), the processing unit 110 can determine that the preceding vehicle can be changing lanes. In another embodiment, the processing unit 110 can analyze the geometry of the snail trail by comparing the lateral distance traveled along the snail trail to the expected curvature of the snail trail. The expected curvature radius can be determined according to the calculation: (δ z 2 + δ x 2 ) / 2 / (δ x ), where δ xrepresents the lateral distance traveled, and δ z represents the longitudinal distance traveled. If the difference between the lateral distance traveled and the expected curvature exceeds a predetermined threshold (e.g., 500 meters to 700 meters), the processing unit 110 can determine that the preceding vehicle is likely changing lanes. In another embodiment, the processing unit 110 can analyze the position of the preceding vehicle. If the position of the preceding vehicle obscures the road polynomial (e.g., the preceding vehicle is overlaid on the road polynomial), the processing unit 110 can determine that the preceding vehicle is likely changing lanes. In cases where the position of the preceding vehicle is such that another vehicle is detected ahead of the preceding vehicle and the snail trails of the two vehicles are not parallel, the processing unit 110 can determine that the (closer) preceding vehicle is likely changing lanes.

[0166] At step 584, the processing unit 110 can determine whether the preceding vehicle 200 is changing lanes based on the analysis performed at step 582. For example, the processing unit 110 can make the determination based on a weighted average of the various analyses performed at step 582. In such an arrangement, for example, a decision by the processing unit 110 that the preceding vehicle is likely changing lanes based on a particular type of analysis can be assigned a value of “1” (and “0” for a determination that the preceding vehicle is not likely changing lanes). Different analyses performed at step 582 can be assigned different weights, and the disclosed embodiments are not limited to any particular combination of analyses and weights.

[0167] Figure 6 is a flowchart illustrating an exemplary process 600 for analyzing stereoscopic images to cause one or more navigational responses in accordance with the disclosed embodiments. At step 610, the processing unit 110 can receive a first and second plurality of images via the data interface 128. For example, cameras included in the image acquisition unit 120 (such as the image capture device 122 having the field of view 202 and the image capture device 124 having the field of view 204) can capture a first and second plurality of images of the area ahead of the vehicle 200 and transmit them to the processing unit 100 over a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, the processing unit 110 can receive the first and second plurality of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0168] At step 620, processing unit 110 can execute stereo image analysis module 404 to perform stereo image analysis on the first and second pluralities of images to create a 3D map of the road ahead of the vehicle and to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, and so on. Stereo image analysis can be performed in a manner similar to that described above in connection with step 218 of FIG. 2. Figures 5A-5D The described steps can be performed in a manner similar to that described above. For example, processing unit 110 can execute stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road markings, traffic lights, road hazards, etc.) within the first and second pluralities of images, filter out a subset of the candidate objects based on various criteria, and perform multi-frame analysis, construct measurements, and determine confidence levels for the remaining candidate objects. In performing the above steps, processing unit 110 can consider information from both the first and second pluralities of images, rather than just information from one set of images. For example, processing unit 110 can analyze differences in pixel-level data (or other subset of data from among the two captured image streams) for candidate objects that appear in both the first and second pluralities of images. As another example, processing unit 110 can estimate the position and / or velocity of a candidate object (e.g., relative to vehicle 200) by observing that the object appears in one image but not the other, or other differences that can exist relative to an object appearing in the case of two image streams. For example, position, velocity, and / or acceleration relative to vehicle 200 can be determined based on a trajectory, position, movement characteristics, etc. of features associated with an object appearing in one or both of the image streams.

[0169] At step 630, processing unit 110 can execute navigation response module 408 to cause one or more navigational responses based on the analysis performed at step 620 and as described above in connection with step 220 of FIG. 2. Figure 4 The described techniques cause one or more navigational responses in vehicle 200. The navigational responses can include, for example, a turn, a lane shift, a change in acceleration, a change in speed, braking, and so on. In some embodiments, processing unit 110 can use data derived from the execution of velocity and acceleration module 406 to cause one or more navigational responses. Further, multiple navigational responses can occur simultaneously, in sequence, or in any combination of these ways.

[0170] Figure 7is a flowchart showing an exemplary process 700 for causing one or more navigational responses based on analysis of images of three sets in accordance with the disclosed embodiments. At step 710, the processing unit 110 can receive first, second, and third pluralities of images via the data interface 128. For example, cameras included in the image capture unit 120, such as the image capture device 122 having the field of view 202, the image capture device 124 having the field of view 204, and the image capture device 126 having the field of view 206, can capture the first, second, and third pluralities of images of the area in front of and / or to the side of the vehicle 200 and transmit them to the processing unit 110 over a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, the processing unit 110 can receive the first, second, and third pluralities of images via three or more data interfaces. For example, each of the image capture devices 122, 124, 126 can have an associated data interface for transmitting data to the processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0171] At step 720, the processing unit 110 can analyze the first, second, and third pluralities of images to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, and the like. The analysis can be performed in a manner similar to the steps described above in connection with Figures 5A-5D and Figure 6 For example, the processing unit 110 can perform monocular image analysis on each of the first, second, and third pluralities of images (e.g., via the execution of the monocular image analysis module 402 and based on the steps described above in connection with Figures 5A-5D Alternatively, the processing unit 110 can perform stereo image analysis on the first and second pluralities of images, the second and third pluralities of images, and / or the first and third pluralities of images (e.g., via the execution of the stereo image analysis module 404 and based on the steps described above in connection with Figure 6 The processed information corresponding to the analysis of the first, second, and / or third pluralities of images can be combined. In some embodiments, the processing unit 110 can perform a combination of monocular image analysis and stereo image analysis. For example, the processing unit 110 can perform monocular image analysis on the first plurality of images (e.g., via the execution of the monocular image analysis module 402) and stereo image analysis on the second and third pluralities of images (e.g., via the execution of the stereo image analysis module 404). The configuration of the image capture devices 122, 124, and 126, including their respective positions and fields of view 202, 204, and 206, can influence the type of analysis performed on the first, second, and third pluralities of images. The disclosed embodiments are not limited to a particular configuration of the image capture devices 122, 124, and 126, or the type of analysis performed on the first, second, and third pluralities of images.

[0172] In some embodiments, processing unit 110 can perform a test on system 100 based on the images captured and analyzed at steps 710 and 720. Such a test can provide an indicator of the overall performance of system 100 for certain configurations of image capture devices 122, 124, and 126. For example, processing unit 110 can determine a ratio of "miss hits" (e.g., instances in which system 100 falsely determines the presence of a vehicle or pedestrian) to "hits."

[0173] At step 730, processing unit 110 can cause one or more navigational responses in vehicle 200 based on information derived from two of the first, second, and third pluralities of images. The selection of two of the first, second, and third pluralities of images can depend on various factors, such as, for example, the number, type, and size of objects detected in each of the pluralities of images. Processing unit 110 can also make the selection based on image quality and resolution, effective field of view reflected in the images, the number of captured frames, the extent to which one or more objects of interest actually appear in the frames (e.g., the percentage of frames in which the object appears, the proportion of the object that appears in each such frame, etc.), and the like.

[0174] In some embodiments, processing unit 110 can select information derived from two of the first, second, and third pluralities of images by determining the degree to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 can combine processed information derived from each of image capture devices 122, 124, and 126 (whether through monocular analysis, stereo analysis, or any combination of the two) and determine visual indicators (e.g., lane markings, detected vehicles and their locations and / or paths, detected traffic lights, etc.) that are consistent across images captured from each of image capture devices 122, 124, and 126. Processing unit 110 can also exclude information that is inconsistent across captured images (e.g., a vehicle changing lanes, a lane model indicating a vehicle that is too close to vehicle 200, etc.). Accordingly, processing unit 110 can select information derived from two of the first, second, and third pluralities of images based on the determination of consistent and inconsistent information.

[0175] The navigational responses can include, for example, a turn, a lane shift, a change in acceleration, and the like. Processing unit 110 can cause the navigational responses based on the analysis performed at step 720 and as described above in connection with FIG. 6. Figure 4The described technology can cause one or more navigational responses. The processing unit 110 can also use data derived from execution of the speed and acceleration module 406 to cause one or more navigational responses. In some embodiments, the processing unit 110 can cause one or more navigational responses based on a relative position, a relative speed, and / or a relative acceleration between the vehicle 200 and an object detected within any of the first, second, and third plurality of images. The plurality of navigational responses can occur simultaneously, in sequence, or in any combination of the above.

[0176] Sparse Road Model for Autonomous Vehicle Navigation

[0177] In some embodiments, the disclosed systems and methods can use sparse maps for autonomous vehicle navigation. In particular, sparse maps can be used for autonomous vehicle navigation along road segments. For example, sparse maps can provide sufficient information for navigating autonomous vehicles without the need to store and / or update large amounts of data. As discussed in further detail below, autonomous vehicles can use sparse maps to navigate one or more roads based on one or more stored trajectories.

[0178] Sparse Map for Autonomous Vehicle Navigation

[0179] In some embodiments, the disclosed systems and methods can generate sparse maps for autonomous vehicle navigation. For example, sparse maps can provide sufficient information for navigation without requiring excessive data storage or data transfer rates. As discussed in further detail below, vehicles (which can be autonomous vehicles) can use sparse maps to navigate one or more roads. For example, in some embodiments, sparse maps can include data related to roads and potential landmarks along roads that can be sufficient for vehicle navigation, but which also presents a small data footprint. For example, sparse data maps described in detail below can require significantly less storage space and data transfer bandwidth as compared to digital maps that include detailed map information, such as image data collected along roads.

[0180] For example, sparse data maps can store three-dimensional polynomial representations of preferred vehicle paths along a road, rather than storing detailed representations of road segments. These paths can require little data storage space. Further, in the described sparse data maps, landmarks can be identified and included in the sparse map road models to assist in navigation. These landmarks can be positioned at any spacing suitable for vehicle navigation, but in some cases, such landmarks need not be identified and included in the model at high density and short spacing. Rather, in some cases, navigation based on landmarks spaced at least 50 meters apart, at least 100 meters apart, at least 500 meters apart, at least 1 kilometer apart, or at least 2 kilometers apart can be possible. As will be discussed in further detail in other sections, sparse maps can be generated based on data collected or measured by vehicles equipped with various sensors and devices, such as image capture devices, global positioning system sensors, motion sensors, etc., as the vehicles travel along a road. In some cases, sparse maps can be generated based on data collected during multiple drives of one or more vehicles along a particular road. Generating sparse maps using multiple drives of one or more vehicles can be referred to as "crowdsourcing" the sparse map.

[0181] Consistent with the disclosed embodiments, autonomous vehicle systems can use sparse maps for navigation. For example, the disclosed systems and methods can distribute sparse maps for generating road navigation models for autonomous vehicles, and can use the sparse maps and / or generated road navigation models to navigate autonomous vehicles along road segments. Sparse maps consistent with the present disclosure can include one or more three-dimensional profiles, which can represent predetermined trajectories that autonomous vehicles can traverse as they move along an associated road segment.

[0182] Sparse maps consistent with the present disclosure can also include data representing one or more road features. Such road features can include identified landmarks, road signature profiles, and any other road-related features that can be useful in navigating a vehicle. Sparse maps consistent with the present disclosure can enable autonomous navigation of vehicles based on a relatively small amount of data included in the sparse map. For example, rather than including detailed representations of a road, such as road edges, road curvatures, images associated with a road segment, or data detailing other physical features associated with a road segment, the disclosed embodiments of sparse maps can require relatively less storage space (and, when portions of the sparse map are transmitted to a vehicle, relatively less bandwidth) but can still adequately provide for autonomous vehicle navigation. The small data footprint of the disclosed sparse maps discussed in further detail below can be achieved in some embodiments by storing representations of road-related elements that require a small amount of data but can still enable autonomous navigation.

[0183] For example, rather than storing a detailed representation of various aspects of a road, the disclosed sparse map can store a polynomial representation of one or more trajectories that a vehicle can follow along a road. Thus, rather than storing (or having to transmit) details about the physical properties of a road, using the disclosed sparse map, a vehicle can be navigated along a particular road segment, in some cases, without having to account for the physical aspects of the road by aligning its travel path with a trajectory (e.g., a polynomial spline) along the particular road segment. In this way, a vehicle can be navigated primarily based on the stored trajectories (e.g., polynomial splines), which can require much less storage space compared to approaches involving storage of road images, road parameters, road layouts, etc.

[0184] In addition to the stored polynomial representation of a trajectory along a road segment, the disclosed sparse map can also include small data objects that can represent road features. In some embodiments, the small data objects can include a digital signature that is derived from a digital image (or digital signal) obtained from a sensor (e.g., a camera or other sensor, such as a suspension sensor) that is on-board a vehicle that travels along a road segment. The digital signature can have a reduced size relative to the signal captured by the sensor. In some embodiments, the digital signature can be created to be compatible with a classifier function that is configured to detect and identify road features from signals captured by the sensor, e.g., during a subsequent drive. In some embodiments, the digital signature can be created such that the digital signature has as small a footprint as possible while maintaining the ability to correlate or match road features based on images captured by a camera on-board a vehicle that travels along the same road segment at a subsequent time (or digital signals generated by the sensor, if the stored signature is not based on images and / or includes other data) to the stored signature.

[0185] In some embodiments, the size of the data object can be further associated with the uniqueness of the road feature. For example, for a road feature that can be detected by a camera on-board a vehicle, and where the camera system on-board the vehicle is coupled to a classifier that is capable of distinguishing image data corresponding to that road feature as being associated with a particular type of road feature (e.g., a road sign), and where such road sign is locally unique in the area (e.g., there are no identical road signs or road signs of the same type in the vicinity), it can be sufficient to store data indicating the type of road feature and its location.

[0186] As will be discussed in further detail below, road features (e.g., landmarks along a road segment) can be stored as small data objects that can represent the road features in relatively few bytes, while providing sufficient information for identifying and using such features for navigation. In one example, a road sign can be identified as an identified landmark that a vehicle's navigation can be based on. A representation of the road sign can be stored in a sparse map to include, for example, data for a few bytes indicating a type of the landmark (e.g., a stop sign) and data for a few bytes indicating a location (e.g., coordinates) of the landmark. Navigation based on such data-light representations of landmarks (e.g., using representations sufficient to locate, identify, and navigate based on landmarks) can provide a desired level of navigational functionality associated with a sparse map, without significantly increasing data overhead associated with the sparse map. Such lean representations of landmarks (and other road features) can take advantage of included sensors and processors on-board such vehicles that are configured to detect, identify, and / or classify certain road features.

[0187] For example, when a sign or a particular type of sign is locally unique within a given area (e.g., when there are no other signs or no other signs of the same type), a sparse map can use data indicating a type of the landmark (sign or particular type of sign), and during navigation (e.g., autonomous navigation), when a camera on-board an autonomous vehicle captures an image of an area including the sign (or particular type of sign), a processor can process the image, detect the sign (if it is indeed present in the image), classify the image as the sign (or particular type of sign), and correlate a location of the image with a location of the sign stored in the sparse map.

[0188] Generating a Sparse Map

[0189] In some embodiments, a sparse map can include at least one line representation of a road surface feature extending along a road segment and a plurality of landmarks associated with the road segment. In certain aspects, a sparse map can be generated via "crowdsourcing," for example, through image analysis of a plurality of images collected as one or more vehicles traverse a road segment.

[0190] Figure 8A sparse map 800 that one or more vehicles (e.g., vehicle 200, which can be an autonomous vehicle) can access to provide autonomous vehicle navigation is shown. Sparse map 800 can be stored in a memory, such as memory 140 or 150. Such memory devices can include any type of non-transitory storage device or computer-readable medium. For example, in some embodiments, memory 140 or 150 can include a hard disk drive, a compact disk, a flash memory, a magnetic-based memory device, an optical-based memory device, etc. In some embodiments, sparse map 800 can be stored in a database (e.g., map database 160), which can be stored in memory 140 or 150 or other types of storage devices.

[0191] In some embodiments, sparse map 800 can be stored on a storage device or non-transitory computer-readable medium provided on-board vehicle 200 (e.g., a storage device included in a navigation system on-board vehicle 200). A processor provided on vehicle 200 (e.g., processing unit 110) can access sparse map 800 stored in the storage device or computer-readable medium provided on-board vehicle 200 in order to generate navigation instructions for guiding autonomous vehicle 200 as the vehicle traverses a road segment.

[0192] However, sparse map 800 need not be stored locally with respect to the vehicle. In some embodiments, sparse map 800 can be stored on a storage device or computer-readable medium provided on a remote server that is in communication with vehicle 200 or a device associated with vehicle 200. A processor provided on vehicle 200 (e.g., processing unit 110) can receive data included in sparse map 800 from the remote server and can execute the data for guiding autonomous driving of vehicle 200. In such embodiments, the remote server can store all or only a portion of sparse map 800. Accordingly, a storage device or computer-readable medium on-board vehicle 200 and / or one or more additional vehicles can store the remaining portion(s) of sparse map 800.

[0193] Further, in such embodiments, sparse map 800 can be made accessible to a plurality of vehicles (e.g., tens, hundreds, thousands, or millions of vehicles, etc.) traversing various road segments. It should also be noted that sparse map 800 can include a plurality of sub-maps. For example, in some embodiments, sparse map 800 can include hundreds, thousands, millions, or more sub-maps that can be used in navigating a vehicle. Such sub-maps can be referred to as local maps, and a vehicle traveling along a road can access any number of local maps relevant to the location in which the vehicle is traveling. The local map portions of sparse map 800 can be stored with global navigation satellite system (GNSS) keys that serve as an index to the database of sparse map 800. Thus, while the calculations for steering angles for navigating a host vehicle in the present system can be performed without reliance on GNSS locations, road features, or landmarks of the host vehicle, such GNSS information can be used to retrieve the relevant local maps.

[0194] Generally, sparse map 800 can be generated based on data collected from one or more vehicles as they travel along roads. For example, using sensors (e.g., cameras, speedometers, GPS, accelerometers, etc.) on-board one or more vehicles, a trajectory of the one or more vehicles traveling along a road can be recorded, and a polynomial representation of a preferred trajectory for a vehicle to follow along the road can be determined based on the collected trajectory of the one or more vehicles traveling. Similarly, data collected by the one or more vehicles can assist in identifying potential landmarks along a particular road. Data collected from passing vehicles can also be used to identify road profile information, such as road width profiles, road roughness profiles, traffic lane spacing profiles, road conditions, etc. Using the collected information, sparse map 800 can be generated and distributed (e.g., for local storage or via on-the-fly data transmission) for use in navigating one or more autonomous vehicles. However, in some embodiments, map generation can not end at the initial generation of the map. As will be discussed in greater detail below, sparse map 800 can be continuously or periodically updated based on data collected from vehicles as those vehicles continue to traverse roads included in sparse map 800.

[0195] The data recorded in sparse map 800 can include location information based on global positioning system (GPS) data. For example, location information can be included in sparse map 800 for various map elements, including, for example, landmark locations, road profile locations, etc. The locations of map elements included in sparse map 800 can be obtained using GPS data collected from vehicles traversing the road. For example, a vehicle passing by a landmark that is identified can determine the location of the identified landmark using GPS location information associated with the vehicle, and a determination of the location of the identified landmark relative to the vehicle (e.g., based on image analysis of data collected from one or more cameras on-board the vehicle). Such location determinations for the identified landmark (or any other feature included in sparse map 800) can be repeated as additional vehicles pass by the location of the identified landmark. The determinations can be used to refine the location information stored in sparse map 800 relative to the identified landmark. For example, in some embodiments, multiple location measurements relative to a particular feature stored in sparse map 800 can be averaged together. However, any other mathematical operation can also be used to refine the stored location of a map element based on multiple determined locations of the map element.

[0196] The sparse maps of the disclosed embodiments can enable autonomous navigation of vehicles using a relatively small amount of stored data. In some embodiments, sparse map 800 can have a data density (e.g., including data representing target trajectories, landmarks, and any other stored road features) of less than 2 MB per kilometer of road, less than 1 MB per kilometer of road, less than 500 kB per kilometer of road, or less than 100 kB per kilometer of road. In some embodiments, the data density of sparse map 800 can be less than 10 kB per kilometer of road, or even less than 2 kB per kilometer of road (e.g., 1.6 kB per kilometer), or no more than 10 kB per kilometer of road, or no more than 20 kB per kilometer of road. In some embodiments, most, if not all, of the roads in the United States can be autonomously navigated using a sparse map having a total of 4 GB or less of data. These data density values can represent average values across the entire sparse map 800, on local maps within sparse map 800, and / or on particular road segments within sparse map 800.

[0197] As noted, sparse map 800 can include representations of multiple target trajectories 810 for guiding autonomous driving or navigation along a road segment. Such target trajectories can be stored as three-dimensional splines. For example, a target trajectory stored in sparse map 800 can be determined based on two or more reconstructed trajectories of previous traversals along a particular road segment for a vehicle. A road segment can be associated with a single target trajectory or multiple target trajectories. For example, on a two-lane road, a first target trajectory can be stored to represent an intended path of travel along the road in a first direction, and a second target trajectory can be stored to represent an intended path of travel along the road in another direction (e.g., opposite the first direction). Additional target trajectories can be stored for a particular road segment. For example, on a multi-lane road, one or more target trajectories can be stored that represent intended paths of travel for vehicles in one or more lanes associated with the multi-lane road. In some embodiments, each lane of a multi-lane road can be associated with its own target trajectory. In other embodiments, there can be fewer stored target trajectories than lanes present on a multi-lane road. In such cases, a vehicle navigating the multi-lane road can use any of the stored target trajectories to guide its navigation by taking into account an amount of lane offset from the lane for which a target trajectory is stored (e.g., if a vehicle is traveling on the leftmost lane of a three-lane highway and a target trajectory is stored only for the middle lane of the highway, the vehicle can use the target trajectory for the middle lane to navigate by taking into account the amount of lane offset between the middle lane and the leftmost lane when generating navigation instructions).

[0198] In some embodiments, a target trajectory can represent an ideal path that a vehicle should take when traveling. For example, a target trajectory can be located at an approximate center of a travel lane. In other cases, a target trajectory can be located at other locations relative to a road segment. For example, a target trajectory can approximately coincide with a center of a road, an edge of a road, or an edge of a lane, among others. In such cases, navigation based on a target trajectory can include a determined offset amount to maintain relative to a location of the target trajectory. Further, in some embodiments, the determined offset amount to maintain relative to a location of a target trajectory can differ based on a type of vehicle (e.g., a passenger vehicle including two axles can have a different offset amount along at least a portion of a target trajectory than a truck including more than two axles).

[0199] The sparse map 800 can also include data related to a plurality of predetermined landmarks 820 associated with particular road segments, local maps, etc. As discussed in greater detail below, these landmarks can be used in the navigation of autonomous vehicles. For example, in some embodiments, the landmarks can be used to determine a current position of a vehicle relative to a stored target trajectory. With this position information, the autonomous vehicle can be able to adjust a heading to match a direction of the target trajectory at the determined position.

[0200] The plurality of landmarks 820 can be identified and stored in the sparse map 800 at any suitable interval. In some embodiments, the landmarks can be stored at a relatively high density (e.g., every few meters or more). However, in some embodiments, significantly larger landmark spacing values can be employed. For example, in the sparse map 800, the identified (or recognized) landmarks can be spaced 10 meters, 20 meters, 50 meters, 100 meters, 1 kilometer, or 2 kilometers apart. In some cases, the identified landmarks can be located at distances spaced even greater than 2 kilometers apart.

[0201] Between landmarks, and thus between multiple determinations of a vehicle's position relative to a target trajectory, the vehicle can navigate based on dead reckoning, in which the vehicle uses sensors to determine its own motion and estimate its position relative to the target trajectory. As errors can accumulate during navigation by dead reckoning, the position determinations relative to the target trajectory can become less and less accurate over time. The vehicle can use the landmarks (and their known positions) present in the sparse map 800 to eliminate errors in the position determinations caused by dead reckoning. In this way, the identified landmarks included in the sparse map 800 can serve as navigation anchors through which an accurate position of the vehicle relative to the target trajectory can be determined. As a certain amount of error is acceptable in position localization, the recognized landmarks need not always be available to an autonomous vehicle. Rather, as noted above, suitable navigation is possible even based on a landmark spacing of 10 meters, 20 meters, 50 meters, 100 meters, 500 meters, 1 kilometer, 2 kilometers, or more. In some embodiments, a density of 1 identified landmark per 1 km of road can be sufficient to maintain longitudinal position determination accuracy to within 1 m. Thus, not every potential landmark appearing along a road segment need be stored in the sparse map 800.

[0202] Further, in some embodiments, lane markings can be used to localize the vehicle during landmark spacing. By using lane markings during landmark spacing, accumulation during navigation by dead reckoning can be minimized.

[0203] In addition to the target trajectory and the identified landmarks, the sparse map 800 can also include information related to various other road features. For example,Figure 9A A representation of a curve along a specific road segment is shown that can be stored in a sparse map 800. In some embodiments, a single lane of the road can be modeled using a three-dimensional polynomial description of the left and right sides of the road. Figure 9A The diagram shows such polynomials representing the left and right sides of a single lane. Regardless of how many lanes a road might have, the road can follow a pattern similar to... Figure 9A The method shown is represented using polynomials. For example, the left and right sides of a multi-lane road can be represented by something similar to... Figure 9A The polynomials shown are represented by polynomials, and the middle lane markings included on multi-lane roads (e.g., dashed lines indicating lane boundaries, solid yellow lines indicating boundaries between lanes traveling in different directions, etc.) can also be represented using, for example... Figure 9A The polynomial shown is represented by a polynomial.

[0204] like Figure 9A As shown, lane 900 can be represented using a polynomial (e.g., a first-order, second-order, third-order, or any suitable order polynomial). For illustration, lane 900 is shown as a two-dimensional lane, and the polynomial is shown as a two-dimensional polynomial. Figure 9A As depicted, lane 900 includes a left side 910 and a right side 920. In some embodiments, more than one polynomial may be used to represent the position of each side of the road or lane boundary. For example, each side of the left side 910 and right side 920 may be represented by multiple polynomials of any suitable length. In some cases, the polynomials may have a length of about 100 m, but other lengths greater or less than 100 m may also be used. Additionally, the polynomials may overlap each other to facilitate a seamless transition when the main vehicle is traveling along the road and navigating based on subsequently encountered polynomials. For example, each side of the left side 910 and right side 920 may be represented by multiple third-order polynomials divided into segments of approximately 100 (an example of a first predetermined range) and overlapping each other by approximately 50 meters. The polynomials representing the left side 910 and right side 920 may have the same order or may not have the same order. For example, in some embodiments, some polynomials may be second-order polynomials, some may be third-order polynomials, and some may be fourth-order polynomials.

[0205] exist Figure 9AIn the illustrated example, the left side 910 of the lane 900 is represented by two sets of third order polynomials. The first set includes polynomial segments 911, 912, and 913. The second set includes polynomial segments 914, 915, and 916. The two sets, while substantially parallel to each other, follow their respective locations on the side of the roadway. The polynomial segments 911, 912, 913, 914, 915, and 916 have a length of approximately 100 meters and overlap the adjacent segments in the series by approximately 50 meters. However, as noted previously, polynomials of different lengths and different amounts of overlap can be used. For example, the polynomials can have lengths of 500 meters, 1 kilometer, or more, and the amount of overlap can vary from 0 to 50 meters, from 50 meters to 100 meters, or to more than 100 meters. Additionally, while the polynomials are illustrated as representing a curve extending in 2D space (e.g., on paper), it should be understood that the polynomials can represent a curve extending in three dimensions (e.g., including a height component) to represent elevation changes in the roadway segment in addition to X-Y curvature. In some embodiments, the polynomials can represent a curve extending in 2D space and a curve extending in three dimensions. Figure 9A Figure 9A In the illustrated example, the right side 920 of the lane 900 is further represented by a first set having polynomial segments 921, 922, and 923 and a second set having polynomial segments 924, 925, and 926.

[0206] Returning to the target trajectories of the sparse map 800, Figure 9B Three-dimensional polynomials representing target trajectories for vehicles to travel along particular roadway segments are illustrated. The target trajectories represent not only the X-Y path that a host vehicle should travel along a particular roadway segment, but also the elevation changes that the host vehicle will experience when traveling along the roadway segment. Thus, each target trajectory in the sparse map 800 can be represented by one or more three-dimensional polynomials, such as the three-dimensional polynomial 950 illustrated in FIG. 9B. The sparse map 800 can include a plurality of target trajectories (e.g., millions or billions or more to represent target trajectories for vehicles along various roadway segments of roads throughout the world). In some embodiments, each target trajectory can correspond to a spline connecting three-dimensional polynomial segments. Figure 9B

[0207] With respect to the data footprint of the polynomial curves stored in the sparse map 800, in some embodiments, each cubic polynomial can be represented by four parameters, each requiring four bytes of data. A suitable representation can be obtained using cubic polynomials that require approximately 192 bytes of data for each 100 m. For a host vehicle traveling at approximately 100 km / hr, this translates to approximately 200 kB / hour in terms of data usage / transmission requirements.

[0208] ​​Sparse map 800 can use a combination of geometric descriptors and metadata to describe the lane network. Geometries can be described by polynomials or splines as described above. Metadata can describe the number of lanes, special features such as shared lanes, and possibly other sparse labels. The total footprint of such indicators can be negligible.

[0209] Accordingly, a sparse map according to embodiments of the present application can include at least one line representation of a road surface feature extending along a road segment, each line representation representing a path along the road segment that substantially corresponds to the road surface feature. In some embodiments, as discussed above, the at least one line representation of the road surface feature can include a spline, a polynomial representation, or a curve. Further, in some embodiments, the road surface feature can include at least one of a road edge or a lane marking. Further, as discussed below with respect to "crowdsourcing," the road surface feature can be identified through image analysis of a plurality of images captured as one or more vehicles traverse the road segment.

[0210] As previously mentioned, sparse map 800 can include a plurality of predetermined landmarks associated with a road segment. Rather than storing actual images of the landmarks and relying on, for example, image recognition analysis based on captured images and stored images, each landmark in sparse map 800 can be represented and identified using less data than the actual images stored. The data representing the landmarks can still include sufficient information to describe or identify the landmarks along the road. Storing data describing the characteristics of the landmarks rather than actual images of the landmarks can reduce the size of sparse map 800.

[0211] Figure 10 Examples of the types of landmarks that can be represented in sparse map 800 are shown. Landmarks can include any visible and identifiable object along a road segment. Landmarks can be selected such that they are fixed and do not frequently change with respect to their location and / or content. The landmarks included in sparse map 800 can be useful in determining the location of vehicle 200 relative to a target trajectory as the vehicle traverses a particular road segment. Examples of landmarks can include traffic signs, directional signs, general signs (e.g., rectangular signs), roadside fixtures (e.g., light poles, reflectors, etc.), and any other suitable category. In some embodiments, lane markings on the road can also be included as landmarks in sparse map 800.

[0212] Figure 10Examples of landmarks shown include traffic signs, directional signs, roadside fixtures, and general signs. Traffic signs can include, for example, speed limit signs (e.g., speed limit sign 1000), yield signs (e.g., yield sign 1005), route number signs (e.g., route number sign 1010), traffic light signs (e.g., traffic light sign 1015), stop signs (e.g., stop sign 1020). Directional signs can include signs that contain one or more arrows indicating one or more directions to different places. For example, directional signs can include: highway signs 1025 with arrows for directing vehicles to different roads or places; exit signs 1030 with arrows directing vehicles to exit a road, and the like. Accordingly, at least one of the plurality of landmarks can include a road sign.

[0213] General signs can be unrelated to traffic. For example, general signs can include billboards for advertising, or welcome signs adjacent to a border between two countries, states, counties, cities, or towns. Figure 10 A general sign 1040 (‘Joe’s Restaurant”) is shown. Although general sign 1040 can have a rectangular shape, as shown, Figure 10 general sign 1040 can have other shapes, such as a square, a circle, a triangle, and the like.

[0214] Landmarks can also include roadside fixtures. Roadside fixtures can be objects that are not signs, and can also be unrelated to traffic or directions. For example, roadside fixtures can include light poles (e.g., light pole 1035), utility poles, traffic light poles, and the like.

[0215] Landmarks can also include beacons that can be specifically designed for use in autonomous vehicle navigation systems. For example, such beacons can include standalone structures placed at predetermined intervals to assist in navigation of host vehicles. Such beacons can also include visual / graphical information added to existing road signs (e.g., icons, emblems, bar codes, and the like) that can be identified or recognized by vehicles traveling along a road segment. Such beacons can also include electronic components. In such embodiments, electronic beacons (e.g., RFID tags, and the like) can be used to communicate non-visual information to host vehicles. Such information can include, for example, landmark identification and / or landmark location information that can be used by host vehicles in determining their location along a target trajectory.

[0216] In some embodiments, landmarks included in sparse map 800 can be represented by data objects of a predetermined size. The data representing a landmark can include any suitable parameters for identifying a particular landmark. For example, in some embodiments, a landmark stored in sparse map 800 can include parameters such as: a physical size of the landmark (e.g., to support estimating a distance to the landmark based on a known size / scale), a distance to a previous landmark, a lateral offset, a height, a type code (e.g., a landmark type - what type of directional sign, traffic sign, etc.), GPS coordinates (e.g., to support global positioning), and any other suitable parameters. Each parameter can be associated with a data size. For example, a landmark size can be stored using 8 bytes of data. A distance to a previous landmark, a lateral offset, and a height can be specified using 12 bytes of data. A type code associated with a landmark, such as a directional sign or a traffic sign, can require about 2 bytes of data. For a general sign, an image signature implementing identification of a general sign can be stored using 50 bytes of data storage. A landmark GPS location can be associated with 16 bytes of data storage. These data sizes for each parameter are merely examples, and other data sizes can also be used.

[0217] Representing landmarks in sparse map 800 in this manner can provide a lean solution for efficiently representing landmarks in a database. In some embodiments, signs can be referred to as semantic signs and non-semantic signs. Semantic signs can include any category of sign for which there is a standardized meaning (e.g., speed limit signs, warning signs, directional signs, etc.). Non-semantic signs can include any sign that is not associated with a standardized meaning (e.g., a general advertising sign, a sign identifying a business establishment, etc.). For example, each non-standard semantic sign can be represented with 38 bytes of data (e.g., 8 bytes for size; 12 bytes for distance to previous landmark, lateral offset, and height; 2 bytes for type code; and 16 bytes for GPS coordinates). Sparse map 800 can use a system of tags to represent landmark types. In some cases, each traffic sign or directional sign can be associated with its own tag, which can be stored in the database as part of the landmark identification. For example, the database can include on the order of 1000 different tags to represent various traffic signs, and on the order of about 10000 different tags to represent directional signs. Of course, any suitable number of tags can be used, and additional tags can be created as needed. In some embodiments, a general sign can be represented using less than about 100 bytes (e.g., about 86 bytes, including 8 bytes for size; 12 bytes for distance to previous landmark, lateral offset, and height; 50 bytes for an image signature; and 16 bytes for GPS coordinates).

[0218] Thus, for semantic signs that do not require image signatures, the data density impact on sparse map 800 can be on the order of 760 bytes per kilometer (e.g., 20 landmarks per kilometer x 38 bytes per landmark = 760 bytes), even with the relatively high landmark density of approximately 1 landmark per 50 meters. Even for general signs that include an image signature component, the data density impact is approximately 1.72 kB per kilometer (e.g., 20 landmarks per kilometer x 86 bytes per landmark = 1720 bytes). For semantic road signs, this equates to approximately 76 kB of data usage per hour for a vehicle traveling at 100 km / hr. For general signs, this equates to approximately 170 kB per hour for a vehicle traveling at 100 km / hr.

[0219] In some embodiments, generally rectangular objects, such as rectangular signs, can be represented in sparse map 800 by no more than 100 bytes of data. The representation of a generally rectangular object (e.g., general sign 1040) in sparse map 800 can include a condensed image signature (e.g., condensed image signature 1045) associated with the generally rectangular object. Such a condensed image signature can be used, for example, to assist in identifying general signs as, for example, recognized landmarks. Such a condensed image signature (e.g., image information derived from actual image data representing the object) can avoid the need to store actual images of the object or the need for comparative image analysis of actual images to be performed to identify landmarks.

[0220] With reference to Figure 10 , sparse map 800 can include or store a condensed image signature 1045 associated with general sign 1040, rather than an actual image of general sign 1040. For example, after an image capture device (e.g., image capture devices 122, 124, or 126) captures an image of general sign 1040, a processor (e.g., image processor 190 or any other processor that can process images on a host vehicle or remotely located relative to a host vehicle) can perform image analysis to extract / create condensed image signature 1045 that includes a unique signature or pattern associated with general sign 1040. In one embodiment, condensed image signature 1045 can include a shape, a color pattern, a brightness pattern, or any other feature that can be extracted from an image of general sign 1040 for use in describing general sign 1040.

[0221] For example, in Figure 10In the condensed image signature 1045, the circles, triangles, and stars can represent areas of different colors. Patterns represented by circles, triangles, and stars can be stored in the sparse map 800, for example, within the 50 bytes designated to include the image signature. It is important to note that circles, triangles, and stars are not necessarily intended to indicate that such shapes are stored as part of the image signature. Rather, these shapes are intended to conceptually represent identifiable areas with discernible color differences, textured areas, graphic shapes, or other variations of characteristics that can be associated with a general sign. Such condensed image signatures can be used to identify landmarks in the form of general signs. For example, condensed image signatures can be used to perform same-not-same analysis based on a comparison of the stored condensed image signature with image data captured, for example, using camera data on an autonomous vehicle.

[0222] Accordingly, multiple landmarks can be identified through image analysis of multiple images captured when one or more vehicles pass through a road segment. As explained below regarding “crowdsourcing,” in some embodiments, the image analysis used to identify multiple landmarks may include accepting a potential landmark when the ratio of images in which the landmark actually appears to images in which the landmark does not appear exceeds a threshold. Furthermore, in some embodiments, the image analysis used to identify multiple landmarks may include rejecting a potential landmark when the ratio of images in which the landmark does not appear to images in which the landmark does appear exceeds a threshold.

[0223] Returning to the main mode of transportation can be used to navigate a target trajectory on a specific road segment. Figure 11A The diagram illustrates a polynomial representation of a trajectory captured during the process of constructing or maintaining a sparse map 800. The polynomial representation of a target trajectory included in the sparse map 800 can be determined based on two or more reconstructed trajectories previously traversed by a vehicle along the same road segment. In some embodiments, the polynomial representation of the target trajectory included in the sparse map 800 can be an aggregation of two or more reconstructed trajectories previously traversed by a vehicle along the same road segment. In some embodiments, the polynomial representation of the target trajectory included in the sparse map 800 can be the average of two or more reconstructed trajectories previously traversed by a vehicle along the same road segment. Other mathematical operations can also be used to construct a target trajectory along a road path based on reconstructed trajectories collected from vehicles traversing the road segment.

[0224] like Figure 11AAs shown, road segment 1100 can be traveled by several vehicles 200 at different times. Each vehicle 200 can collect data related to the path taken by the vehicle along the road segment. The path traveled by a particular vehicle can be determined based on camera data, accelerometer information, speed sensor information, and / or GPS information, among other potential sources. Such data can be used to reconstruct the trajectory of the vehicle along the road segment, and based on these reconstructed trajectories, a target trajectory (or multiple target trajectories) for the particular road segment can be determined. Such target trajectories can represent a preferred path for a host vehicle (e.g., guided by an autonomous navigation system) as the vehicle travels along the road segment.

[0225] In Figure 11A In the example shown, a first reconstructed trajectory 1101 can be determined based on data received from a first vehicle traversing road segment 1100 at a first time period (e.g., day 1), a second reconstructed trajectory 1102 can be obtained from a second vehicle traversing road segment 1100 at a second time period (e.g., day 2), and a third reconstructed trajectory 1103 can be obtained from a third vehicle traversing road segment 1100 at a third time period (e.g., day 3). Each trajectory 1101, 1102, and 1103 can be represented by a polynomial, such as a three-dimensional polynomial. It should be noted that in some embodiments, any of the reconstructed trajectories can be assembled on-board a vehicle traversing road segment 1100.

[0226] Additionally or alternatively, such reconstructed trajectories can be determined on a server-side based on information received from vehicles traversing road segment 1100. For example, in some embodiments, vehicles 200 can transmit data related to their motion along road segment 1100 (e.g., steering angle, heading, time, location, speed, sensed road geometry, and / or sensed landmarks, etc.) to one or more servers. The servers can reconstruct a trajectory for vehicles 200 based on the received data. The servers can also generate a target trajectory for guiding autonomous vehicles to navigate, which will later travel along the same road segment 1100 based on the first trajectory 1101, the second trajectory 1102, and the third trajectory 1103. While a target trajectory can be associated with a single previous traversal of a road segment, in some embodiments, each target trajectory included in sparse map 800 can be determined based on two or more reconstructed trajectories of vehicles traversing the same road segment. In Figure 11AIn this context, the target trajectory is represented by 1110. In some embodiments, the target trajectory 1110 can be generated based on the average value of the first trajectory 1101, the second trajectory 1102, and the third trajectory 1103. In some embodiments, the target trajectory 1110 included in the sparse map 800 can be an aggregation (e.g., a weighted combination) of two or more reconstructed trajectories.

[0227] Figure 11B and Figure 11C The concept of a target trajectory associated with a road segment existing within geographic region 1111 is further illustrated. For example... Figure 11B As shown, a first road segment 1120 within geographic region 1111 may include a multi-lane road comprising two lanes 1122 designated for vehicles traveling in a first direction and two additional lanes 1124 designated for vehicles traveling in a second direction opposite to the first direction. Lanes 1122 and lanes 1124 may be separated by double yellow lines 1123. Geographic region 1111 may also include branch road segments 1130 intersecting with road segment 1120. Road segment 1130 may include a two-lane road, with each lane designated for a different direction of travel. Geographic region 1111 may also include other road features such as stop lines 1132, stop signs 1134, speed limit signs 1136, and hazard signs 1138.

[0228] like Figure 11CAs shown, sparse map 800 can include a local map 1140 that includes a road model for assisting autonomous navigation of vehicles within geographic region 1111. For example, local map 1140 can include target trajectories for one or more lanes associated with road segments 1120 and / or 1130 within geographic region 1111. For example, local map 1140 can include target trajectories 1141 and / or 1142 that an autonomous vehicle can access or rely upon when traversing lane 1122. Similarly, local map 1140 can include target trajectories 1143 and / or 1144 that an autonomous vehicle can access or rely upon when traversing lane 1124. Further, local map 1140 can include target trajectories 1145 and / or 1146 that an autonomous vehicle can access or rely upon when traversing road segment 1130. Target trajectory 1147 represents a preferred path that an autonomous vehicle should follow when transitioning from lane 1120 (specifically, relative to target trajectory 1141 associated with the rightmost lane in lane 1120) to road segment 1130 (specifically, relative to target trajectory 1145 associated with the first side of road segment 1130). Similarly, target trajectory 1148 represents a preferred path that an autonomous vehicle should follow when transitioning from road segment 1130 (specifically, relative to target trajectory 1146) to a portion of road segment 1124 (specifically, as shown, relative to target trajectory 1143 associated with the left lane in lane 1124).

[0229] Sparse map 800 can also include representations of other road-related features associated with geographic region 1111. For example, sparse map 800 can also include representations of one or more landmarks identified in geographic region 1111. Such landmarks can include a first landmark 1150 associated with stop line 1132, a second landmark 1152 associated with stop sign 1134, a third landmark associated with speed limit sign 1154, and a fourth landmark 1156 associated with hazard sign 1138. For example, such landmarks can be used to assist an autonomous vehicle in determining its current position relative to any of the target trajectories shown, so that the vehicle can adjust its heading to match the direction of the target trajectory at the determined position.

[0230] In some embodiments, sparse map 800 can also include road characteristic profiles. Such road characteristic profiles can be associated with any discernable / measurable changes in at least one parameter associated with a road. For example, in some cases, such profiles can be associated with changes in road surface information, such as changes in surface roughness for a particular road segment, changes in road width on a particular road segment, changes in distance between dashed lines drawn along a particular road segment, changes in road curvature along a particular road segment, etc. Figure 11DAn example of a road characteristic profile 1160 is shown. While the profile 1160 can represent any of the parameters mentioned above or other parameters, in one example, the profile 1160 can represent a measure of road surface roughness as obtained, for example, by monitoring one or more sensors that provide an output indicative of an amount of suspension displacement when a vehicle is traveling a particular road segment.

[0231] Alternatively or concurrently, the profile 1160 can represent a variation in road width as determined based on image data obtained via a camera on-board a vehicle traveling a particular road segment. Such a profile can be useful, for example, in determining a particular position of an autonomous vehicle relative to a particular target trajectory. That is, as the autonomous vehicle traverses a road segment, it can measure a profile associated with one or more parameters associated with that road segment. If the measured profile can be correlated / matched with a predetermined profile that maps variations in the parameter with respect to position along the road segment, the measured profile and the predetermined profile can be used (e.g., by superimposing respective portions of the measured and predetermined profiles) to determine a current position along the road segment, and thereby a current position relative to a target trajectory for the road segment.

[0232] In some embodiments, the sparse map 800 can include different trajectories based on different characteristics associated with users of autonomous vehicles, environmental conditions, and / or other parameters related to driving. For example, in some embodiments, different trajectories can be generated based on different user preferences and / or profiles. The sparse map 800 including such different trajectories can be provided to different autonomous vehicles of different users. For example, some users can prefer to avoid toll roads, while other users can prefer to select the shortest or fastest route regardless of whether a toll road is on that route. The disclosed system can generate different sparse maps with different trajectories based on such different user preferences or profiles. As another example, some users can prefer to drive on fast moving lanes, while other users can prefer to always maintain a position in the center lane.

[0233] Different trajectories can be generated and included in the sparse map 800 based on different environmental conditions, such as day and night, snow, rain, fog, etc. Autonomous vehicles driving in different environmental conditions can be provided with sparse maps 800 generated based on such different environmental conditions. In some embodiments, cameras provided on autonomous vehicles can detect environmental conditions, and such information can be provided back to the server that generates and provides the sparse maps. For example, the server can generate or update a sparse map 800 that has already been generated to include trajectories that can be more suitable or safer for autonomous driving in the detected environmental conditions. The updating of the sparse map 800 based on environmental conditions can be performed dynamically as the autonomous vehicle is driving along a road.

[0234] Other different parameters related to driving can also be used as a basis for generating different sparse maps and providing them to different autonomous vehicles. For example, when an autonomous vehicle is traveling at a high rate of speed, turns can be more difficult. Trajectories associated with a particular lane rather than a road can be included in sparse map 800 such that when a vehicle follows a particular trajectory, the autonomous vehicle can be maintained within a particular lane. When images captured by a camera on-board the autonomous vehicle indicate that the vehicle has drifted outside of the lane (e.g., crossed a lane marker), an action can be triggered within the vehicle to bring the vehicle back to the designated lane according to the particular trajectory.

[0235] Crowdsourcing a Sparse Map

[0236] In some embodiments, the disclosed systems and methods can generate sparse maps for autonomous vehicle navigation. For example, the disclosed systems and methods can use crowd-sourced data to generate a sparse, which one or more autonomous vehicles can use to navigate along a road system. As used herein, "crowd-sourcing" means receiving data from various vehicles (e.g., autonomous vehicles) that travel over a road segment at different times, and such data is used to generate and / or update a road model. The model can in turn be communicated to vehicles or other vehicles that subsequently travel along the road segment to assist autonomous vehicles in navigation. The road model can include a plurality of target trajectories that represent preferred trajectories that autonomous vehicles should follow as they traverse the road segment. The target trajectories can be the same as reconstructed actual trajectories collected from vehicles that traversed the road segment, which can be communicated from the vehicles to a server. In some embodiments, the target trajectories can be different from actual trajectories previously taken by one or more vehicles as they traversed the road segment. The target trajectories can be generated based on the actual trajectories (e.g., by averaging or any other suitable operation).

[0237] The vehicle trajectory data that a vehicle can upload to a server can correspond to an actual reconstructed trajectory of the vehicle, or can correspond to a recommended trajectory that can be based on or related to the actual reconstructed trajectory of the vehicle, but can be different from the actual reconstructed trajectory. For example, a vehicle can modify its actual reconstructed trajectory and submit (e.g., recommend) the modified actual trajectory to a server. The road model can use the recommended, modified trajectory as a target trajectory for autonomous navigation by other vehicles.

[0238] In addition to trajectory information, other information potentially used in constructing the sparse data map 800 can include information related to potential landmark candidates. For example, through crowdsourcing of information, the disclosed systems and methods can identify potential landmarks in an environment and refine landmark locations. Landmarks can be used by a navigation system of an autonomous vehicle to determine and / or adjust a position of the vehicle along a target trajectory.

[0239] A reconstructed trajectory that a vehicle can generate as the vehicle travels along a road can be obtained by any suitable method. In some embodiments, a reconstructed trajectory can be formed by stitching together motion segments of the vehicle using, for example, ego-motion estimation (e.g., three-dimensional translation and three-dimensional rotation of the camera and thus the body of the vehicle). Rotation and translation estimates can be determined based on analysis of images captured by one or more image capture devices along with information from other sensors or devices, such as inertial sensors and velocity sensors. For example, inertial sensors can include accelerometers or other suitable sensors configured to measure changes in translation and / or rotation of the body of the vehicle. The vehicle can include velocity sensors that measure a velocity of the vehicle.

[0240] In some embodiments, ego-motion of the camera (and thus the body of the vehicle) can be estimated based on optical flow analysis of captured images. Optical flow analysis of a sequence of images identifies motion of pixels from the sequence of images and determines motion of the vehicle based on the identified motion. Ego-motion can be integrated over time and along a road segment to reconstruct a trajectory associated with a road segment that the vehicle has followed.

[0241] Data (e.g., reconstructed trajectories) collected by different vehicles at different times in multiple drives along a road segment can be used to construct road models (e.g., including target trajectories, etc.) included in the sparse data map 800. Data collected by different vehicles at different times in multiple drives along a road segment can also be averaged to improve accuracy of the models. In some embodiments, data regarding road geometry and / or landmarks can be received from multiple vehicles that travel through a common road segment at different times. Such data received from different vehicles can be combined to generate and / or update road models.

[0242] The geometry of the reconstructed trajectory (as well as the target trajectory) along a road segment can be represented by a curve in three-dimensional space, which can be a spline connecting three-dimensional polynomials. The reconstructed trajectory curve can be determined from analysis of video streams or multiple images captured by a camera mounted on a vehicle. In some embodiments, a position a few meters ahead of the current position of the vehicle is identified in each frame or image. This position is the position to which the vehicle is expected to travel in a predetermined time period. This operation can be repeated frame by frame, and at the same time, the vehicle can compute ego motion (rotation and translation) of the camera. At each frame or image, a short-range model for the desired path is generated by the vehicle in a reference frame attached to the camera. The short-range models can be stitched together to obtain a three-dimensional model of the road in a certain coordinate system, which can be an arbitrary or predetermined coordinate system. The three-dimensional model of the road can then be fitted by a spline, which can include or connect one or more polynomials of suitable order.

[0243] To summarize a short-range road model at each frame, one or more detection modules can be used. For example, a bottom-up lane detection module can be used. The bottom-up lane detection module can be useful when lane markings are painted on the road. This module can find edges in the image and assemble them together to form lane markings. A second module can be used with the bottom-up lane detection module. The second module is an end-to-end deep neural network that can be trained to predict the correct short-range path from an input image. In both modules, the road model can be detected in the image coordinate system and transformed to three-dimensional space that can be virtually attached to the camera.

[0244] Although the reconstructed trajectory modeling approach can introduce accumulation of errors (which can include noise components) due to the integration of ego motion over long time periods, such errors can be insignificant because the generated model can provide sufficient accuracy for navigation on a local scale. Furthermore, it is possible to eliminate the integrated errors by using external sources of information such as satellite images or geodetic measurements. For example, the disclosed systems and methods can use a GNSS receiver to eliminate accumulated errors. However, GNSS positioning signals can not always be available and accurate. The disclosed systems and methods can implement a maneuver application that is weakly dependent on the availability and accuracy of GNSS positioning. In such systems, the use of GNSS signals can be limited. For example, in some embodiments, the disclosed systems can use GNSS signals only for database indexing purposes.

[0245] In some embodiments, the range scale (e.g., local scale) that can be relevant for autonomous vehicle navigation maneuvers can be on the order of 50 meters, 100 meters, 200 meters, 300 meters, etc. Such distances can be used because the geometric road model is primarily used for two purposes: planning a forward trajectory, and localizing the vehicle on the road model. In some embodiments, when the control algorithm maneuvers the vehicle according to a target point that is located 1.3 seconds (or any other time, such as 1.5 seconds, 1.7 seconds, 2 seconds, etc.) ahead, the planning task can use the model for a typical range of 40 meters ahead (or any other suitable distance ahead, such as 20 meters, 30 meters, 50 meters). The localization task uses the road model for a typical range of 60 meters behind the vehicle (or any other suitable distance, such as 50 meters, 100 meters, 150 meters, etc.) according to a method called "tail alignment" described in more detail in another section. The disclosed systems and methods can generate a geometric model that is sufficiently accurate for a particular range (such as 100 meters) so that the planned trajectory will not deviate from the center of the lane by more than, for example, 30 cm.

[0246] As explained above, a three-dimensional road model can be constructed by detecting short-range segments and stitching them together. The stitching can be accomplished by computing a six-degree ego-motion model using video and / or images captured by a camera, data from inertial sensors that reflect vehicle motion, and a host vehicle speed signal. The cumulative error can be small enough for a certain local range scale (such as on the order of 100 meters). All of this can be done in a single drive on a particular road segment.

[0247] In some embodiments, multiple drives can be used to average the resulting model and further improve its accuracy. The same car can drive the same route multiple times, or multiple cars can send their collected model data to a central server. In any case, a matching process can be performed to identify overlapping models and implement averaging to generate a target trajectory. Once convergence criteria are met, the constructed model (e.g., including the target trajectory) can be used for maneuvering. Subsequent drives can be used for further model refinement to adapt to infrastructure changes.

[0248] If multiple cars are connected to a central server, it becomes feasible to share drive experiences (such as sensed data) among multiple cars. Each vehicle client can store a partial copy of a global road model that can be relevant for its current location. A bidirectional update process between the vehicles and the server can be performed by the vehicles and the server. The small footprint concept discussed above enables the disclosed systems and methods to use very small bandwidth for the bidirectional update.

[0249] Information related to the potential landmark can also be determined and forwarded to the central server. For example, the disclosed systems and methods can determine one or more physical properties of the potential landmark based on one or more images that include the landmark. The physical properties can include a physical size (e.g., height, width) of the landmark, a distance from the vehicle to the landmark, a distance from the landmark to a previous landmark, a lateral position of the landmark (e.g., a position of the landmark relative to a lane of travel), a GPS coordinate of the landmark, a type of the landmark, an identification of text on the landmark, etc. For example, the vehicle can analyze one or more images captured by the camera to detect a potential landmark, such as a speed limit sign.

[0250] The vehicle can determine a distance from the vehicle to the landmark based on the analysis of the one or more images. In some embodiments, the distance can be determined based on an analysis of an image of the landmark using a suitable image analysis method, such as a scaling method and / or an optical flow method. In some embodiments, the disclosed systems and methods can be configured to determine a type or classification of the potential landmark. In cases where the vehicle determines that a certain potential landmark corresponds to a predetermined type or classification stored in the sparse map, it can be sufficient for the vehicle to communicate an indication of the type or classification of the landmark to the server along with its location. The server can store such indications. At a later time, other vehicles can capture an image of the landmark, process the image (e.g., using a classifier), and compare the results from processing the image to the indication of the type of the landmark stored in the server. There can be various types of landmarks, and different types of landmarks can be associated with different types of data to be uploaded to the server and stored in the server, different processing on-board the vehicles can detect the landmarks and communicate information about the landmarks to the server, and systems on-board the vehicles can receive landmark data from the server and use the landmark data to identify the landmarks in autonomous navigation.

[0251] In some embodiments, multiple autonomous vehicles traveling on a road segment can communicate with a server. The vehicles (or clients) can generate curves describing their drives in an arbitrary coordinate system (e.g., by ego-motion integration). The vehicles can detect landmarks and localize these landmarks in the same frame. The vehicles can upload the curves and landmarks to the server. The server can collect data from the vehicles for multiple drives and generate a unified road model. For example, as discussed below with respect to Figure 19 The server can use the uploaded curves and landmarks to generate a sparse map with a unified road model.

[0252] The server can also distribute the model to clients (e.g., vehicles). For example, the server can distribute a sparse map to one or more vehicles. The server can update the model continuously or periodically as it receives new data from the vehicles. For example, the server can process the new data to assess whether it contains information that should trigger an updated model or the creation of new data on the server. The server can then distribute the updated model or updates to the vehicles to provide autonomous vehicle navigation.

[0253] The server can use one or more criteria to determine whether new data received from a vehicle should trigger an update to the model or the creation of new data. For example, when new data indicates that a previously identified landmark at a specific location no longer exists or has been replaced by another landmark, the server can determine that the new data should trigger an update to the model. As another example, when new data indicates that a road segment has been closed, and this has been confirmed by data received from other vehicles, the server can determine that the new data should trigger an update to the model.

[0254] The server can distribute the updated model (or an updated portion of the model) to one or more vehicles traveling on a road segment that are associated with the model update. The server can also distribute the updated model to vehicles about to travel on a road segment, or to vehicles whose planned journeys include the road segment associated with the model update. For example, if an autonomous vehicle is traveling along another road segment before reaching the road segment associated with the update, the server can distribute the updated or modified model to the autonomous vehicle before it reaches that road segment.

[0255] In some embodiments, a remote server can collect tracks and landmarks from multiple clients (e.g., vehicles traveling along a common road segment). The server can use landmarks to match curves and create an average road model based on the tracks collected from the multiple vehicles. The server can also calculate the road map and the most probable path at each node or connection of the road segment. For example, the remote server can align tracks to generate a crowdsourced sparse map from the collected tracks.

[0256] The server can average landmark attributes (such as distances between one landmark and another (e.g., previous landmarks along the road segment) received from multiple vehicles traveling along a common road segment to determine arc length parameters and support path-by-path positioning and speed calibration for each customer vehicle. The server can average the physical dimensions of landmarks measured by multiple vehicles traveling along a common road segment and identifying the same landmarks. The average physical dimensions can be used to support distance estimation, such as the distance from a vehicle to a landmark. The server can average the lateral position of landmarks (e.g., from the lane a vehicle is about to enter to the location of the landmark) measured by multiple vehicles traveling along a common road segment and identifying the same landmarks. The average lateral position can be used to support lane assignment. The server can average the GPS coordinates of landmarks measured by multiple vehicles traveling along the same road segment and identifying the same landmarks. The average GPS coordinates of landmarks can be used to support global positioning or placement of landmarks in the road model.

[0257] In some embodiments, the server can identify model changes based on data received from vehicles, such as construction, detours, new signs, sign removal, etc. Upon receiving new data from vehicles, the server can update the model continuously, periodically, or instantly. The server can distribute the updated model or the updated model to vehicles to provide autonomous navigation. For example, as discussed further below, the server can use crowdsourced data to filter out “ghost” landmarks detected by vehicles.

[0258] In some embodiments, the server can analyze driver interventions during autonomous driving. The server can analyze data received from the vehicle at the time and location of the intervention, and / or data received prior to the intervention. The server can identify portions of the data that led to or is closely related to the intervention, such as data indicating temporary lane closures or pedestrians in the road. The server can update the model based on the identified data. For example, the server can modify one or more trajectories stored in the model.

[0259] Figure 12 This is a schematic diagram of a system that uses crowdsourcing to generate sparse maps (and uses crowdsourced sparse maps for distribution and navigation). Figure 12 Road segment 1200, comprising one or more lanes, is shown. Multiple vehicles 1205, 1210, 1215, 1220, and 1225 may travel on road segment 1200 at the same time or at different times (although...). Figure 12(As shown in the diagram, they appear on road segment 1200 at the same time). At least one of vehicles 1205, 1210, 1215, 1220, and 1225 can be an autonomous vehicle. For the simplicity of this example, it is assumed that all vehicles 1205, 1210, 1215, 1220, and 1225 are autonomous vehicles.

[0260] Each vehicle may be similar to a vehicle disclosed in other embodiments (e.g., vehicle 200) and may include components or devices included or associated with vehicles disclosed in other embodiments. Each vehicle may be equipped with an image capture device or camera (e.g., image capture device 122 or camera 122). As indicated by the dashed lines, each vehicle may communicate with a remote server 1230 via wireless communication path 1235, via one or more networks (e.g., via cellular networks and / or the Internet, etc.). Each vehicle may transmit data to and receive data from server 1230. For example, server 1230 may collect data from multiple vehicles traveling on road segment 1200 at different times and may process the collected data to generate an autonomous vehicle road navigation model or an update to the model. Server 1230 may transmit the autonomous vehicle road navigation model or an update to the model to the vehicle that transmitted data to server 1230. Server 1230 may later transmit the autonomous vehicle road navigation model or an update to the model to other vehicles traveling on road segment 1200.

[0261] When vehicles 1205, 1210, 1215, 1220, and 1225 travel on road segment 1200, navigation information collected (e.g., detected, sensed, or measured) by vehicles 1205, 1210, 1215, 1220, and 1225 can be transmitted to server 1230. In some embodiments, the navigation information may be associated with public road segment 1200. The navigation information may include a trajectory associated with each of vehicles 1205, 1210, 1215, 1220, and 1225 as each travels on road segment 1200. In some embodiments, the trajectory may be reconstructed based on data sensed by various sensors and devices provided on vehicle 1205. For example, the trajectory may be reconstructed based on at least one of accelerometer data, rate data, landmark data, road geometry or profile data, vehicle positioning data, and self-motion data. In some embodiments, the trajectory can be reconstructed based on data from inertial sensors (such as accelerometers) and the speed of vehicle 1205 sensed by a rate sensor. Furthermore, in some embodiments, the trajectory can be determined based on the sensed ego motion of a camera (e.g., by a processor onboard each of vehicles 1205, 1210, 1215, 1220, and 1225), which may indicate three-dimensional translation and / or three-dimensional rotation (or rotational motion). The ego motion of the camera (and therefore the vehicle body) can be determined by analyzing one or more images captured by the camera.

[0262] In some embodiments, the trajectory of vehicle 1205 may be determined by a processor provided on vehicle 1205 and transmitted to server 1230. In other embodiments, server 1230 may receive data sensed by various sensors and devices provided in vehicle 1205 and determine the trajectory based on the data received from vehicle 1205.

[0263] In some embodiments, navigation information transmitted from vehicles 1205, 1210, 1215, 1220, and 1225 to server 1230 may include data about the road surface, road geometry, or road overview. The geometry of road segment 1200 may include lane structure and / or landmarks. Lane structure may include the total number of lanes in road segment 1200, lane types (e.g., one-way lanes, two-way lanes, driving lanes, overtaking lanes, etc.), lane markings, lane widths, etc. In some embodiments, navigation information may include lane assignment, such as which lane a vehicle is entering among multiple lanes. For example, lane assignment may be associated with the value "3," which indicates that the vehicle is traveling in the third lane from the left or right. As another example, lane assignment may be associated with the text value "center lane," indicating that the vehicle is traveling in the center lane.

[0264] Server 1230 may store navigation information on a non-transient computer-readable medium, such as a hard disk drive, compact disk, magnetic tape, memory, etc. Server 1230 may (e.g., via a processor included in server 1230) generate at least a portion of an autonomous vehicle road navigation model for a common road segment 1200 based on navigation information received from multiple vehicles 1205, 1210, 1215, 1220, and 1225, and may store this model as part of a sparse map. Server 1230 may determine a trajectory associated with each lane based on crowdsourced data (e.g., navigation information) received from multiple vehicles (e.g., 1205, 1210, 1215, 1220, and 1225) traveling in lanes of the road segment at different times. Server 1230 may generate an autonomous vehicle road navigation model or a portion of a model (e.g., an updated portion) based on multiple trajectories determined based on the crowdsourced navigation data. Server 1230 may transmit a model or an updated portion of a model to one or more of the autonomous vehicles 1205, 1210, 1215, 1220, and 1225 traveling on road segment 1200, or to any other autonomous vehicle later traveling on the road segment, to update the existing autonomous vehicle road navigation model provided in the vehicle's navigation system. The autonomous vehicle road navigation model can be used by the autonomous vehicles when navigating autonomously along the common road segment 1200.

[0265] As explained above, autonomous vehicle road navigation models can be included in sparse maps (e.g., Figure 8The sparse map 800 depicted in the image is used to guide autonomous vehicles. The sparse map 800 may include sparse records of data relating to road geometry and / or landmarks along the road lines, which can provide sufficient information for guiding autonomous vehicle navigation without requiring excessive data storage. In some embodiments, the autonomous vehicle road navigation model may be stored separately from the sparse map 800, and map data from the sparse map 800 may be used when modeling for navigation. In some embodiments, the autonomous vehicle road navigation model may use the map data included in the sparse map 800 to determine a target trajectory along road segment 1200 to guide autonomous vehicles 1205, 1210, 1215, 1220, and 1225, or subsequent vehicles traveling along road segment 1200. For example, when the autonomous vehicle road navigation model is executed by a processor included in the navigation system of vehicle 1205, the model can enable the processor to compare the trajectory determined based on navigation information received from vehicle 1205 with a predetermined trajectory included in sparse map 800 to verify and / or correct the current driving route of vehicle 1205.

[0266] In autonomous vehicle road navigation models, the geometry of road features or target trajectories can be encoded using curves in three-dimensional space. In one embodiment, the curve can be a three-dimensional spline comprising one or more connected three-dimensional polynomials. As understood by those skilled in the art, a spline can be a numerical function defined piecewise by a series of polynomials used to fit the data. The spline used to fit the three-dimensional geometry data of the road can include linear splines (first order), quadratic splines (second order), cubic splines (third order), or any other splines (other orders), or combinations thereof. A spline can include one or more three-dimensional polynomials of different orders connecting (e.g., fitting) data points of the three-dimensional geometry data of the road. In some embodiments, the autonomous vehicle road navigation model can include a three-dimensional spline corresponding to a target trajectory along a common road segment (e.g., road segment 1200) or a lane of road segment 1200.

[0267] As explained above, the autonomous vehicle road navigation model included in the sparse map may include other information, such as the identification of at least one landmark along road segment 1200. This landmark may be visible within the field of view of a camera (e.g., camera 122) mounted on each of the vehicles 1205, 1210, 1215, 1220, and 1225. In some embodiments, camera 122 may capture an image of the landmark. A processor provided on vehicle 1205 (e.g., processor 180, 190, or processing unit 110) may process the image of the landmark to extract landmark identification information. The landmark identification information, rather than the actual image of the landmark, may be stored in the sparse map 800. The landmark identification information may require significantly less storage space than the actual image. Other sensors or systems (e.g., a GPS system) may also provide some identification information for the landmark (e.g., the location of the landmark). Landmarks may include at least one of the following: traffic signs, arrow markings, lane markings, dashed lane markings, traffic lights, stop lines, directional markings (e.g., highway exit signs with arrows indicating directions, highway signs with arrows pointing in different directions or locations), landmark beacons, or light poles. A landmark beacon refers to a device (e.g., an RFID device) installed along a road segment that sends or reflects signals to a receiver installed on a vehicle, such that when the vehicle passes the device, the beacon received by the vehicle and the device's location (e.g., determined from the device's GPS location) can be used as a landmark to be included in the autonomous vehicle road navigation model and / or sparse map 800.

[0268] The identification of at least one landmark may include the location of at least one landmark. The location of the landmark may be determined based on location measurements performed using sensor systems (e.g., GPS, inertial positioning systems, landmark beacons, etc.) associated with multiple vehicles 1205, 1210, 1215, 1220, and 1225. In some embodiments, the location of the landmark may be determined by averaging location measurements detected, collected, or received by sensor systems on different vehicles 1205, 1210, 1215, 1220, and 1225 during multiple drives. For example, vehicles 1205, 1210, 1215, 1220, and 1225 may transmit location measurement data to server 1230, which may average the location measurements and use the averaged location measurements as the location of the landmark. The location of the landmark may be continuously refined using measurements received from vehicles during subsequent drives.

[0269] Landmark identification can include the size of the landmark. A processor provided on a vehicle (e.g., 1205) can estimate the physical size of the landmark based on analysis of the image. Server 1230 can receive multiple estimates of the physical size of the same landmark from different vehicles traveling on different roads. Server 1230 can average the different estimates to arrive at the physical size of the landmark and store this landmark size in a road model. The physical size estimate can be used to further determine or estimate the distance from the vehicle to the landmark. The distance to the landmark can be estimated based on the current speed of the vehicle and the expansion scale, which is based on the position of the landmark in the image relative to the expansion focus of the camera. For example, the distance to the landmark can be estimated by Z = V * dt * R / D, where V is the speed of the vehicle, R is the distance from the landmark to the expansion focus in the image from time t1, and D is the change in distance to the landmark in the image from t1 to t2. dt represents (t2 - t1). For example, the distance to a landmark can be estimated using Z = V * dt * R / D, where V is the speed of the vehicle, R is the distance between the landmark and the extended focal point in the image, dt is the time interval, and D is the image displacement of the landmark along the epipolar line. Other equations equivalent to the above, such as Z = V * ω / Δω, can be used to estimate the distance to a landmark. Here, V is the speed of the vehicle, ω is the image length (similar to the object width), and Δω is the change in the image length per unit time.

[0270] When the physical size of a landmark is known, the distance to the landmark can be determined based on the following equation: Z = f * W / ω, where f is the focal length, W is the size of the landmark (e.g., height or width), and ω is the number of pixels the landmark leaves the image. Using this equation, the change in distance Z can be expressed as ΔZ = f * W * Δω / ω 2 The distance is calculated using +f*ΔW / ω, where ΔW is averaged to zero, and Δω is the number of pixels representing the accuracy of the bounding box in the image. The estimated physical size of the landmark can be calculated by averaging multiple observations on the server side. The error in the distance estimation can be very small. When using the formula above, there are two potential sources of error: ΔW and Δω. Their contribution to the distance error is given by ΔZ = f*W*Δω / ω. 2 +f*ΔW / ω is given. However, ΔW decays to zero by averaging, so ΔZ is determined by Δω (e.g., the inaccuracy of the bounding box in the image).

[0271] For landmarks of unknown size, the distance to the landmark can be estimated by tracking feature points on the landmark between consecutive frames. For example, certain features appearing on a speed limit sign can be tracked between two or more image frames. Based on these tracked features, a per-feature-point distance distribution can be generated. The distance estimate can be extracted from the distance distribution. For example, the distance that appears most frequently in the distance distribution can be used as the distance estimate. As another example, the average of the distance distribution can be used as the distance estimate.

[0272] Figure 13 An example autonomous vehicle road navigation model is shown, represented by multiple three-dimensional splines 1301, 1302 and 1303. Figure 13 The curves 1301, 1302, and 1303 shown are for illustrative purposes only. Each spline may include one or more three-dimensional polynomials connecting multiple data points 1310. Each polynomial may be a first-order polynomial, a second-order polynomial, a third-order polynomial, or any suitable combination of polynomials with different orders. Each data point 1310 may be associated with navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225. In some embodiments, each data point 1310 may be associated with data related to landmarks (e.g., the size, location, and identification information of landmarks) and / or road feature profiles (e.g., road geometry, road roughness profile, road curvature profile, road width profile). In some embodiments, some data points 1310 may be associated with data related to landmarks, while other data points may be associated with data related to road feature profiles.

[0273] Figure 14The diagram illustrates raw location data 1410 (e.g., GPS data) received from five separate driving sessions. A driving session can be separated from another driving session if it is experienced by a single vehicle at the same time, by the same vehicle at separate times, or by a single vehicle at separate times. To account for errors in the location data 1410 and different positions of vehicles within the same lane (e.g., one vehicle might be driving closer to the left side of the lane than another), server 1230 can use one or more statistical techniques to generate a map skeleton 1420 to determine whether the variations in the raw location data 1410 are actual dispersions or statistical errors. Each path within the skeleton 1420 can be linked back to the raw data 1410 that formed that path. For example, the path between A and B within the skeleton 1420 is linked to the raw data 1410 from driving sessions 2, 3, 4, and 5, but not from driving session 1. Estimate 1420 may not be detailed enough for navigating vehicles (e.g., because unlike the splines described above, frame 1420 incorporates driving data from multiple lanes on the same road), but it can provide useful topological information and can be used to define intersections.

[0274] Figure 15 An example is shown that it can generate additional details for a sparse map within segments of a map skeleton (e.g., segments A to B within skeleton 1420). Figure 15 As depicted, data (e.g., self-motion data, road marking data, etc.) can be represented as a function of the driving position S (or S1 or S2). Server 1230 can identify landmarks in a sparse map by identifying unique matches between landmarks 1501, 1503, and 1505 of driving 1510 and landmarks 1507 and 1509 of driving 1520. Such a matching algorithm can result in the identification of landmarks 1511, 1513, and 1515. However, those skilled in the art will recognize that other matching algorithms can be used. For example, probabilistic optimization can be used instead of or in combination with unique matching. Server 1230 can align driving directions longitudinally to align matched landmarks. For example, server 1230 can select one driving direction (e.g., driving 1520) as a reference driving direction and then offset and / or elastically stretch another (multiple) driving direction (e.g., driving 1510) for alignment.

[0275] Figure 16 An example of aligned landmark data used in sparse maps is shown. Figure 16 In the example, landmark 1610 includes road signs. Figure 16 The examples further depict data from multiple drives 1601, 1603, 1605, 1607, 1609, 1611, and 1613. In Figure 16In the example, the data from drive 1613 consists of "ghost" landmarks, and server 1230 can identify it accordingly because none of drives 1601, 1603, 1605, 1607, 1609, and 1611 include an identifier for a landmark near the identified landmark in drive 1613. Accordingly, server 1230 can accept a potential landmark when the ratio of images in which the landmark actually appears to images in which the landmark does not appear exceeds a threshold, and / or can reject a potential landmark when the ratio of images in which the landmark does not appear to images in which the landmark actually appears exceeds a threshold.

[0276] Figure 17 A system 1700 for generating driving data is described, which can be used for crowdsourced sparse maps. For example... Figure 17 As depicted, system 1700 may include a camera 1701 and a positioning device 1703 (e.g., a GPS locator). Camera 1701 and positioning device 1703 may be mounted on a vehicle (e.g., one of vehicles 1205, 1210, 1215, 1220, and 1225). Camera 1701 may generate multiple types of data, such as self-motion data, traffic sign data, road data, etc. Camera data and location data may be segmented into driving segments 1705. For example, driving segments 1705 may each have camera data and location data from a driving distance of less than 1 km.

[0277] In some embodiments, system 1700 may remove redundancy in driving segment 1705. For example, if a landmark appears in multiple images from camera 1701, system 1700 may remove redundant data so that driving segment 1705 contains only the location of the landmark and a copy of any metadata associated with the landmark. As a further example, if lane markings appear in multiple images from camera 1701, system 1700 may remove redundant data so that driving segment 1705 contains only the location of the lane markings and a copy of any metadata associated with the lane markings.

[0278] 1700 also includes a server (e.g., server 1230). Server 1230 can receive driving segments 1705 from the vehicle and recombine the driving segments 1705 into a single driving segment 1707. This arrangement can allow for reduced bandwidth requirements when transferring data between the vehicle and the server, while also allowing the server to store data related to the entire driving process.

[0279] Figure 18 The diagram depicts further configurations for crowdsourced sparse maps. Figure 17 The system. For example... Figure 17As shown, system 1700 includes a vehicle 1810 that captures driving data using, for example, a camera (which generates, for example, self-motion data, traffic sign data, road data, etc.) and a positioning device (e.g., a GPS locator). As in Figure 17 As in the example of the vehicle 1810, the collected data is divided into driving segments (in...). Figure 18 The data is described as "DS1 1", "DS2 1", and "DSN 1" in the diagram. Then, server 1230 receives the driving segments and reconstructs the driving data from the received segments (in...). Figure 18 It is depicted as "Driving 1" in the middle.

[0280] like Figure 18 As further described, system 1700 also receives data from an auxiliary vehicle. For example, vehicle 1820 also uses, for example, cameras (which generate, for example, self-motion data, traffic sign data, road data, etc.) and positioning devices (e.g., GPS locators) to capture driving data. Similar to vehicle 1810, vehicle 1820 divides the collected data into driving segments (in... Figure 18 The text describes the process as "DS1 2", "DS2 2", and "DSN 2". Then, server 1230 receives the road segment and reconstructs the driving sequence from the received segment (in...). Figure 18 (Described as "Driving 2" in the text). Any number of additional vehicles can be used. For example, Figure 18 It also includes "Vehicle N", which captures driving data and segments it into driving segments (in Figure 18 The data is described as "DS1 N", "DS2 N", "DSN N" and sent to server 1230 for reconstruction as driving (in Figure 18 (It is depicted in the middle as "driving N").

[0281] like Figure 18 As depicted, server 1230 can construct a sparse map (described as "map") using reconstructed driving data (e.g., "driving 1", "driving 2", and "driving N") collected from multiple vehicles (e.g., "car 1" (also labeled as vehicle 1810), "car 2" (also labeled as vehicle 1820), and "car N").

[0282] Figure 19 This is a flowchart illustrating an example process 1900 for generating a sparse map for autonomous vehicle navigation along a road segment. Process 1900 can be executed by one or more processing devices included in server 1230.

[0283] Process 1900 may include receiving multiple images captured as one or more vehicles traverse a road segment (step 1905). Server 1230 may receive images from cameras included in one or more of vehicles 1205, 1210, 1215, 1220, and 1225. For example, as vehicle 1205 travels along road segment 1200, camera 122 may capture one or more images of the environment surrounding vehicle 1205. In some embodiments, server 1230 may also receive streamlined image data with redundancy removed by a processor on vehicle 1205, as described above regarding... Figure 17 The discussion.

[0284] Process 1900 may further include: at least one line representation of road surface features extending along a road segment based on multiple image identifiers (step 1910). Each line representation may represent a path along a road segment substantially corresponding to the road surface features. For example, server 1230 may analyze environmental images received from camera 122 to identify road edges or lane markings and determine a trajectory of travel along road segment 1200 associated with such road edges or lane markings. In some embodiments, the trajectory (or line representation) may include splines, polynomial representations, or curves. Server 1230 may determine the trajectory of vehicle 1205 based on camera ego motion (e.g., three-dimensional translation and / or three-dimensional rotational motion) received at step 1905.

[0285] Process 1900 may also include identifying multiple landmarks associated with a road segment based on multiple images (step 1910). For example, server 1230 may analyze environmental images received from camera 122 to identify one or more landmarks, such as road signs along road segment 1200. Server 1230 may use analysis of multiple images acquired as one or more vehicles traverse the road segment to identify landmarks. To enable crowdsourcing, the analysis may include rules regarding the acceptance and rejection of potential landmarks associated with the road segment. For example, the analysis may include: accepting a potential landmark when the ratio of images in which the landmark actually appears to images in which the landmark does not appear exceeds a threshold; and / or rejecting a potential landmark when the ratio of images in which the landmark does not appear to images in which the landmark does appear exceeds a threshold.

[0286] Process 1900 may include other operations or steps performed by server 1230. For example, navigation information may include target trajectories for vehicles traveling along a road segment, and process 1900 may include: server 1230 clustering vehicle trajectories associated with multiple vehicles traveling on the road segment; and determining a target trajectory based on the clustered vehicle trajectories, as discussed in further detail below. Clustering vehicle trajectories may include server 1230 clustering multiple trajectories associated with vehicles traveling on the road segment into multiple clusters based on at least one of the vehicle's absolute heading or the vehicle's lane assignment. Generating a target trajectory may include server 1230 averaging the clustered trajectories. As a further example, process 1900 may include: aligning the data received in step 1905. Other processes or steps performed by server 1230 as described above may also be included in process 1900.

[0287] The disclosed systems and methods may include other features. For example, the disclosed systems may use local coordinates instead of global coordinates. For autonomous driving, some systems may present data in world coordinates. For example, latitude and longitude coordinates on the Earth's surface may be used. To manipulate the vehicle using a map, the primary vehicle can determine its position and orientation relative to the map. To locate the vehicle on the map and to find the rotational transformation between the primary reference frame and the world reference frame (e.g., north, east, and down), it seems natural to use an onboard GPS device. Once the primary reference frame is aligned with the map reference frame, the desired route can be expressed in the primary reference frame, and manipulation commands can be calculated or generated.

[0288] The disclosed systems and methods enable autonomous vehicle navigation (e.g., maneuvering control) with a low-occupancy model, which can be collected by the autonomous vehicle itself without the aid of expensive survey equipment. To support autonomous navigation (e.g., maneuvering applications), the road model can include a sparse map with the geometry of the road, its lane structure, and landmarks, which can be used to determine the vehicle's position or location along a trajectory included in the model. As discussed above, the generation of the sparse map can be performed by a remote server that communicates with and receives data from the vehicle traveling on the road. This data can include sensed data, a trajectory reconstructed based on the sensed data, and / or a recommended trajectory that can represent a modified reconstructed trajectory. As discussed below, the server can transmit the model back to the vehicle or other vehicles subsequently traveling on the road to assist in autonomous navigation.

[0289] Figure 20A block diagram of server 1230 is shown. Server 1230 may include a communication unit 2005, which may include hardware components (e.g., communication control circuitry, switches, and antennas) and software components (e.g., communication protocols, computer code). For example, communication unit 2005 may include at least one network interface. Server 1230 can communicate with vehicles 1205, 1210, 1215, 1220, and 1225 via communication unit 2005. For example, server 1230 can receive navigation information transmitted from vehicles 1205, 1210, 1215, 1220, and 1225 via communication unit 2005. Server 1230 can distribute autonomous vehicle road navigation models to one or more autonomous vehicles via communication unit 2005.

[0290] Server 1230 may include at least one non-transient storage medium 2010, such as a hard disk drive, compact disk, magnetic tape, etc. Storage device 1410 may be configured to store data such as navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225 and / or autonomous vehicle road navigation models generated by server 1230 based on the navigation information. Storage device 2010 may be configured to store any other information, such as sparse maps (e.g., those mentioned above regarding...). Figure 8 The sparse map under discussion (800).

[0291] As an addition to or replacement for storage device 2010, server 1230 may include memory 2015. Memory 2015 may be similar to or different from memory 140 or 150. Memory 2015 may be non-transient memory, such as flash memory, random access memory, etc. Memory 2015 may be configured to store data, such as computer code or instructions executable by a processor (e.g., processor 2020), map data (e.g., data from sparse map 800), autonomous vehicle road navigation models, and / or navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225.

[0292] Server 1230 may include at least one processing device 2020 configured to execute computer code or instructions stored in memory 2015 to perform various functions. For example, processing device 2020 may analyze navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225 and generate an autonomous vehicle road navigation model based on the analysis. Processing device 2020 may control communication unit 1405 to distribute the autonomous vehicle road navigation model to one or more autonomous vehicles (e.g., one or more of vehicles 1205, 1210, 1215, 1220, and 1225, or any vehicle subsequently traveling on road segment 1200). Processing device 2020 may be similar to or different from processor 180, 190, or processing unit 110.

[0293] Figure 21 A block diagram of memory 2015 is shown. Memory 2015 can store computer code or instructions for performing one or more operations to generate a road navigation model used in autonomous vehicle navigation. Figure 21 As shown, memory 2015 may store one or more modules for performing operations for processing vehicle navigation information. For example, memory 2015 may include model generation module 2105 and model distribution module 2110. Processor 2020 may execute instructions stored in any of the modules 2105 and 2110 included in memory 2015.

[0294] The model generation module 2105 may store instructions that, when executed by the processor 2020, may generate at least a portion of an autonomous vehicle road navigation model for a common road segment (e.g., road segment 1200) based on navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225. For example, in generating the autonomous vehicle road navigation model, the processor 2020 may cluster vehicle trajectories along the common road segment 1200 into different clusters. The processor 2020 may determine a target trajectory along the common road segment 1200 based on the clustered vehicle trajectories for each of the different clusters. Such operations may include finding the mean or average trajectory of the clustered vehicle trajectories in each cluster (e.g., by averaging data representing the clustered vehicle trajectories). In some embodiments, the target trajectory may be associated with a single lane of the common road segment 1200.

[0295] Road models and / or sparse maps can store trajectories associated with road segments. These trajectories, referred to as target trajectories, are provided to autonomous vehicles for autonomous navigation. Target trajectories can be received from multiple vehicles, or generated based on actual or recommended trajectories (actual trajectories with some modifications) received from multiple vehicles. The target trajectories included in the road model or sparse map can be continuously updated (e.g., averaged) using new trajectories received from other vehicles.

[0296] Vehicles traveling on a road segment can collect data through various sensors. This data may include landmarks, road feature profiles, vehicle motion (e.g., accelerometer data, speed data), vehicle location (e.g., GPS data), and may reconstruct the actual trajectory itself, or transmit the data to a server that will reconstruct the vehicle's actual trajectory. In some embodiments, the vehicle may transmit data associated with the trajectory (e.g., a curve in any reference frame), landmark data, and lane assignments along the travel path to server 1230. Various vehicles traveling along the same road segment on multiple occasions may have different trajectories. Server 1230 can identify routes or trajectories associated with each lane from the trajectories received from the vehicles through a clustering process.

[0297] Figure 22 A process is illustrated for clustering vehicle trajectories associated with vehicles 1205, 1210, 1215, 1220, and 1225 to determine target trajectories for a common road segment (e.g., road segment 1200). The target trajectories or multiple target trajectories determined according to the clustering process can be included in an autonomous vehicle road navigation model or sparse map 800. In some embodiments, vehicles 1205, 1210, 1215, 1220, and 1225 traveling along road segment 1200 can transmit multiple trajectories 2200 to server 1230. In some embodiments, server 1230 can generate trajectories based on landmark, road geometry, and vehicle motion information received from vehicles 1205, 1210, 1215, 1220, and 1225. To generate an autonomous vehicle road navigation model, server 1230 can cluster vehicle trajectories 1600 into multiple clusters 2205, 2210, 2215, 2220, 2225, and 2230, such as... Figure 22 As shown.

[0298] Clustering can be performed using various criteria. In some embodiments, all drivers in the cluster may be similar in terms of absolute heading along road segment 1200. Absolute heading can be obtained from GPS signals received by vehicles 1205, 1210, 1215, 1220, and 1225. In some embodiments, absolute heading can be obtained using dead reckoning. As those skilled in the art will understand, dead reckoning can be used to determine the current position and thus heading of vehicles 1205, 1210, 1215, 1220, and 1225 using previously determined positions, estimated speeds, etc. Trajectories clustered by absolute heading can be used to identify routes along the road.

[0299] In some embodiments, all drivers in the cluster may have similar lane assignments along road segment 1200 (e.g., in the same lane before and after an intersection). The trajectory of lane assignment clustering can be useful for identifying lanes along the road. In some embodiments, both criteria (e.g., absolute heading and lane assignment) can be used for clustering.

[0300] In each cluster 2205, 2210, 2215, 2220, 2225, and 2230, trajectories can be averaged to obtain a target trajectory associated with a specific cluster. For example, trajectories from multiple drives associated with the same lane cluster can be averaged. The average trajectory can be a target trajectory associated with a specific lane. To average the trajectories across the cluster, server 1230 can choose a reference frame for any trajectory C0. For all other trajectories (C1, ..., Cn), server 1230 can find a rigid transformation mapping Ci to C0, where i = 1, 2, ..., n, where n is a positive integer corresponding to the total number of trajectories included in the cluster. Server 1230 can compute the mean curve or trajectory in the C0 reference frame.

[0301] In some embodiments, landmarks can define arc length matching between different driving modes, which can be used for trajectory-lane alignment. In some embodiments, lane markings before and after intersections can be used for trajectory-lane alignment.

[0302] To assemble lanes from a trajectory, server 1230 can select any reference frame for the lanes. Server 1230 can draw partially overlapping lanes to the selected reference frame. Server 1230 can continue drawing until all lanes are in the same reference frame. Lanes adjacent to each other can be aligned as if they were the same lane, and then these lanes can be offset laterally.

[0303] Landmarks identified along road segments can be plotted to a common reference frame, first at lane level and then at intersection level. For example, the same landmark can be identified multiple times by multiple vehicles during multiple drives. The data received about the same landmark in different drives may differ slightly. Such data can be averaged and plotted to the same reference frame, such as the C0 reference frame. Alternatively, the variance of the data for the same landmark received across multiple drives can be calculated.

[0304] In some embodiments, each lane of road segment 120 may be associated with a target trajectory and certain landmarks. The target trajectory or multiple such target trajectories may be included in an autonomous vehicle road navigation model, which may later be used by other autonomous vehicles traveling along the same road segment 1200. Landmarks identified by vehicles 1205, 1210, 1215, 1220, and 1225 as they travel along road segment 1200 may be recorded in association with the target trajectory. The target trajectory and landmark data may be continuously or periodically updated using new data received from other vehicles during subsequent driving.

[0305] For the localization of autonomous vehicles, the disclosed systems and methods may use an extended Kalman filter. The vehicle's position may be determined based on three-dimensional position data and / or three-dimensional orientation data, by integrating predictions of future positions ahead of the vehicle's current position using self-motion. The vehicle's localization may be corrected or adjusted by image observation of landmarks. For example, when the vehicle detects a landmark within an image captured by a camera, the landmark may be compared with known landmarks stored in a road model or sparse map 800. The known landmarks may have known locations (e.g., GPS data) along a target trajectory stored in the road model and / or sparse map 800. The distance from the vehicle to the landmark can be estimated based on the current speed and the image of the landmark. The vehicle's position along the target trajectory may be adjusted based on the distance to the landmark and the known location of the landmark (stored in the road model or sparse map 800). The location / localization data of landmarks stored in the road model and / or sparse map 800 (e.g., averages from multiple drives) may be assumed to be accurate.

[0306] In some embodiments, the disclosed system can form a closed-loop subsystem, wherein the estimation of the vehicle's six-DOF position (e.g., three-dimensional position data plus three-dimensional orientation data) can be used to navigate the autonomous vehicle (e.g., maneuvering the wheels of the autonomous vehicle) to reach a desired point (e.g., 1.3 seconds ahead of the stored value). Conversely, data from maneuvering and actual navigation measurements can be used to estimate the six-DOF position.

[0307] In some embodiments, roadside markers (such as light poles and utility poles or cable poles) can be used as landmarks for locating vehicles. Other landmarks (such as traffic signs, traffic lights, arrows on the road, stop lines, and static features or characteristics of objects along a road segment) can also be used as landmarks for locating vehicles. When using markers for location, the x-view of the marker (i.e., from the vehicle's perspective) can be used instead of the y-view (i.e., the distance to the marker), because the base of the marker may be obscured, and sometimes they are not on the road surface.

[0308] Figure 23 A navigation system for a vehicle is shown, which can be used for autonomous navigation using a crowdsourced sparse map. For illustration, the vehicle is referred to as vehicle 1205. Figure 23 The vehicles shown can be any other vehicles disclosed herein, including, for example, vehicles 1210, 1215, 1220, and 1225, as well as vehicle 200 shown in other embodiments. Figure 12 As shown, vehicle 1205 can communicate with server 1230. Vehicle 1205 may include image capture device 122 (e.g., camera 122). Vehicle 1205 may include navigation system 2300, configured to provide navigation guidance for vehicle 1205 traveling on a road (e.g., road segment 1200). Vehicle 1205 may also include other sensors, such as rate sensor 2320 and accelerometer 2325. Rate sensor 2320 may be configured to detect the rate of vehicle 1205. Accelerometer 2325 may be configured to detect acceleration or deceleration of vehicle 1205. Figure 23 The vehicle 1205 shown can be an autonomous vehicle, and the navigation system 2300 can be used to provide navigation guidance for autonomous driving. Alternatively, the vehicle 1205 can also be a non-autonomous, manually controlled vehicle, and the navigation system 2300 can still be used to provide navigation guidance.

[0309] The navigation system 2300 may include a communication unit 2305 configured to communicate with the server 1230 via a communication path 1235. The navigation system 2300 may also include a GPS unit 2310 configured to receive and process GPS signals. The navigation system 2300 may further include at least one processor 2315 configured to process data such as GPS signals, map data from a sparse map 800 (which may be stored on a storage device provided on the vehicle 1205 and / or received from the server 1230), road geometry sensed by a road profile sensor 2330, images captured by a camera 122, and / or an autonomous vehicle road navigation model received from the server 1230. The road profile sensor 2330 may include different types of devices for measuring different types of road profiles, such as road surface roughness, road width, road elevation, road curvature, etc. For example, the road profile sensor 2330 may include devices that measure the motion of the suspension of the vehicle 2305 to derive a road roughness profile. In some embodiments, the road profile sensor 2330 may include a radar sensor for measuring the distance from the vehicle 1205 to the roadside (e.g., roadside obstacles), thereby measuring the width of the road. In some embodiments, the road profile sensor 2330 may include devices configured to measure the vertical elevation of the road. In some embodiments, the road profile sensor 2330 may include devices configured to measure the curvature of the road. For example, a camera (e.g., camera 122 or another camera) may be used to capture images of the road showing its curvature. The vehicle 1205 may use such images to detect the road curvature.

[0310] At least one processor 2315 may be programmed to receive at least one environmental image associated with vehicle 1205 from camera 122. At least one processor 2315 may analyze the at least one environmental image to determine navigation information associated with vehicle 1205. The navigation information may include a trajectory associated with the travel of vehicle 1205 along road segment 1200. At least one processor 2315 may determine the trajectory based on the motion of camera 122 (and therefore vehicle) such as three-dimensional translation and three-dimensional rotation. In some embodiments, at least one processor 2315 may determine the translation and rotation of camera 122 based on the analysis of multiple images acquired by camera 122. In some embodiments, the navigation information may include lane assignment information (e.g., where lane vehicle 1205 is traveling along road segment 1200). The autonomous vehicle road navigation model can be generated and / or updated by the server 1230 using navigation information transmitted from the vehicle 1205 to the server 1230. This autonomous vehicle road navigation model can be transmitted back from the server 1230 to the vehicle 1205 to provide autonomous navigation guidance for the vehicle 1205.

[0311] At least one processor 2315 may also be programmed to transmit navigation information from vehicle 1205 to server 1230. In some embodiments, the navigation information may be transmitted to server 1230 along with road information. Road location information may include at least one of the following: GPS signals received by GPS unit 2310; landmark information; road geometry; lane information, etc. At least one processor 2315 may receive an autonomous vehicle road navigation model or a portion thereof from server 1230. The autonomous vehicle road navigation model received from server 1230 may include at least one update based on the navigation information transmitted from vehicle 1205 to server 1230. The portion of the model transmitted from server 1230 to vehicle 1205 may include the updated portion of the model. At least one processor 2315 may induce at least one navigation maneuver (e.g., manipulation, such as turning, braking, acceleration, overtaking another vehicle, etc.) performed by vehicle 1205 based on the received autonomous vehicle road navigation model or the updated portion thereof.

[0312] At least one processor 2315 may be configured to communicate with various sensors and components included in the vehicle 1205, including a communication unit 1705, a GPS unit 2315, a camera 122, a rate sensor 2320, an accelerometer 2325, and a road condition sensor 2330. At least one processor 2315 may collect information or data from the various sensors and components and transmit the information or data to the server 1230 via the communication unit 2305. Alternatively or additionally, the various sensors or components of the vehicle 1205 may also communicate with the server 1230 and transmit data or information collected by the sensors or components to the server 1230.

[0313] In some embodiments, vehicles 1205, 1210, 1215, 1220, and 1225 can communicate with each other and share navigation information, such that at least one of vehicles 1205, 1210, 1215, 1220, and 1225 can generate an autonomous vehicle road navigation model using crowdsourcing (e.g., based on information shared by other vehicles). In some embodiments, vehicles 1205, 1210, 1215, 1220, and 1225 can share navigation information with each other, and each vehicle can update the autonomous vehicle road navigation model provided in its own vehicle. In some embodiments, at least one of vehicles 1205, 1210, 1215, 1220, and 1225 (e.g., vehicle 1205) can function as a central vehicle. At least one processor 2315 of the central vehicle (e.g., vehicle 1205) can perform some or all of the functions performed by server 1230. For example, at least one processor 2315 of the central vehicle can communicate with other vehicles and receive navigation information from them. The at least one processor 2315 of the central vehicle can generate an autonomous vehicle road navigation model or an update to the model based on shared information received from other vehicles. The at least one processor 2315 of the central vehicle can transmit the autonomous vehicle road navigation model or the update to the model to other vehicles to provide autonomous navigation guidance.

[0314] Drawing Lane Markers and Navigation Based on Drawn Lane Markers

[0315] As previously discussed, the autonomous vehicle road navigation model and / or sparse map 800 may include multiple drawn lane markings associated with road segments. These drawn lane markings can be used during autonomous vehicle navigation, as discussed in more detail below. For example, in some embodiments, the drawn lane markings can be used to determine the lateral position and / or orientation relative to a planned trajectory. Using this positional information, the autonomous vehicle may be able to adjust its heading to match the orientation of a target trajectory at the determined location.

[0316] Vehicle 200 can be configured to detect lane markings in a given road segment. The road segment can include any markings on the road used to guide traffic of vehicles on the road. For example, lane markings can be continuous or dashed lines distinguishing the edges of driving lanes. Lane markings can also include double lines, such as double continuous lines, double dashed lines, or a combination of continuous and dashed lines, which, for example, indicate whether overtaking is permitted in adjacent lanes. Lane markings can also include highway entrance and exit markings, such as indicating deceleration lanes for exit ramps or dotted lines indicating that the lane is dedicated to turning or the end of the lane. Markings can further indicate work zones, temporary lane departures, travel paths through intersections, median strips, dedicated lanes (e.g., bicycle lanes, HOV lanes, etc.) or other miscellaneous markings (e.g., pedestrian crossings, speed bumps, railway crossings, stop lines, etc.).

[0317] Vehicle 200 can use a camera to capture images of surrounding lane markings, such as image capture devices 122 and 124 included in image acquisition unit 120. Vehicle 200 can analyze the images based on features identified within one or more captured images to detect point locations associated with lane markings. These point locations can be uploaded to a server to represent lane markings in sparse map 800. Depending on the camera position and field of view, lane markings can be detected simultaneously from both sides of the vehicle from a single image. In other embodiments, different cameras can be used to capture images on multiple sides of the vehicle. Instead of uploading actual images of lane markings, the markings can be stored as splines or a series of points in sparse map 800, thereby reducing the size of sparse map 800 and / or the data that must be remotely uploaded by the vehicle.

[0318] Figures 24A-24D An exemplary point location that can be detected by vehicle 200 to represent a specific lane marking is shown. Similar to the landmarks described above, vehicle 200 can use various image recognition algorithms or software to identify point locations within a captured image. For example, vehicle 200 can identify a series of edge points, corner points, or various other point locations associated with a specific lane marking. Figure 24A A continuous lane marking 2410, detectable by vehicle 200, is shown. Lane marking 2410 may represent the outer edge of a road, indicated by a continuous white line. (Example) Figure 24AAs shown, vehicle 200 can be configured to detect multiple edge location points 2411 along lane markings. Location points 2411 can be collected to represent lane markings at any interval sufficient to create drawn lane markings in a sparse map. For example, lane markings can be represented by one point per meter of detected edges, one point every five meters of detected edges, or other suitable intervals. In some embodiments, the interval can be determined by other factors rather than a set interval, such as, for example, based on the point where vehicle 200 has the highest confidence level of the detected point's location. Although Figure 24A The edge location points on the inner edge of lane marking 2410 are shown, but points can be collected along the outer edge of the line or along both edges. Furthermore, although in Figure 24A The diagram shows a single line, but similar edge points can be detected for double continuous lines. For example, point 2411 can be detected along one or both edges of a continuous line.

[0319] Depending on the type or shape of the lane markings, the vehicle 200 may also represent lane markings in different ways. Figure 24B An exemplary dashed lane marking 2420 that can be detected by vehicle 200 is shown. (As in...) Figure 24A Unlike the way edge points are identified, vehicles can detect a series of corner points 2421 representing the corners of the lane dashed lines to define the complete boundary of the dashed lines. Although Figure 24B Each corner of a given dashed line marker is shown to be located, but vehicle 200 can detect or upload a subset of the points shown in the figure. For example, vehicle 200 can detect the leading edge or leading corner of a given dashed line marker, or it can detect the two corners closest to the interior of the lane. Furthermore, not every dashed line marker can be captured; for example, vehicle 200 can capture and / or record points representing samples of dashed line markers (e.g., every one, every three, every five, etc.) or capture and / or record dashed line markers at predefined intervals (e.g., every meter, every five meters, every ten meters, etc.). Corners can also be detected for similar lane markings, such as those indicating a lane for an exit ramp, a lane where a particular lane is ending, or other various lane markings that may have detectable corners. Corners can also be detected for lane markings consisting of double dashed lines or a combination of continuous lines and dashed lines.

[0320] In some embodiments, the points uploaded to the server to generate the drawn lane markings may represent points other than detected edge points or corner points. Figure 24CA series of points that can represent the centerline of a given lane marking are shown. For example, a continuous lane 2410 can be represented by centerline point 2441 along the centerline 2440 of the lane marking. In some embodiments, the vehicle 200 can be configured to detect these center points using various image recognition techniques, such as convolutional neural networks (CNNs), scale-invariant feature transforms (SIFTs), oriented gradient histogram (HOG) features, or other techniques. Alternatively, the vehicle 200 can detect other points, such as... Figure 24A The edge point 2411 shown is used, and the centerline point 2411 can be calculated, for example, by detecting each edge point and determining the midpoint between the edge points. Similarly, the dashed lane marking 2420 can be represented by the centerline point 2451 along the centerline 2450 of the lane marking. The centerline point can be located at the edge of the dashed line, such as... Figure 24C As shown, or located at various other positions along the centerline. For example, each dash can be represented by a single point at the geometric center of the dash. These points can also be spaced apart along the centerline at predetermined intervals (e.g., every meter, every 5 meters, every 10 meters, etc.). Centerline point 2451 can be detected directly by vehicle 200, or can be based on other detected reference points (such as corner point 2421, etc.). Figure 24B (As shown) to calculate. The center line can also be used to represent other lane marking types, such as double lines, using a similar technique as above.

[0321] In some embodiments, the vehicle 200 may identify points representing other features, such as vertices between two intersecting lane markings. Figure 24D An exemplary point representing the intersection between two lane markings 2460 and 2465 is shown. Vehicle 200 can calculate a vertex 2466 representing the intersection between the two lane markings. For example, one of lane markings 2460 or 2465 may represent a train crossing area or other crossing area within a road segment. Although lane markings 2460 and 2465 are shown intersecting perpendicularly to each other, various other configurations can be detected. For example, lane markings 2460 and 2465 may intersect at other angles, or one or both of the lane markings may terminate at vertex 2466. Similar techniques can also be applied to intersections between dashed lines or other lane marking types. In addition to vertex 2466, various other points 2467 can be detected, providing further information about the orientation of lane markings 2460 and 2465.

[0322] Vehicle 200 can associate real-world coordinates with each detected point of the lane marking. For example, a location identifier can be generated to be uploaded to a server for drawing the lane markings; this location identifier includes the coordinates of each point. The location identifier can further include other identifying information about these points, including whether the point represents a corner, edge, center, etc. Therefore, vehicle 200 can be configured to determine the real-world location of each point based on analysis of an image. For example, vehicle 200 can detect other features in the image, such as the various landmarks described above, to locate the real-world location of the lane markings. This may involve: determining the position of the lane markings in the image relative to detected landmarks; or determining the position of the vehicle based on detected landmarks and then determining the distance from the vehicle (or the vehicle's target trajectory) to the lane markings. When landmarks are unavailable, the position of the lane marking point can be determined relative to the position of the vehicle determined based on dead reckoning. The real-world coordinates included in the location identifier can be represented as absolute coordinates (e.g., latitude / longitude coordinates) or relative to other features, such as based on longitudinal position along the target trajectory and lateral distance from the target trajectory. The location identifiers can then be uploaded to a server to generate drawn lane markings in a navigation model (such as a sparse map 800). In some embodiments, the server can construct splines representing lane markings for road segments. Alternatively, the vehicle 200 can generate splines and upload them to the server for recording in the navigation model.

[0323] Figure 24E An exemplary navigation model or sparse map for a corresponding road segment is shown, which includes drawn lane markings. The sparse map may include a target trajectory 2475 for a vehicle to follow along the road segment. As described above, the target trajectory 2475 may represent the ideal path taken by the vehicle when traveling on the corresponding road segment, or it may be located elsewhere on the road (e.g., the centerline of the road, etc.). The target trajectory 2475 can be calculated using various methods described above, such as aggregation (e.g., weighted combination) of two or more reconstructed trajectories of vehicles traversing the same road segment.

[0324] In some embodiments, a target trajectory can be generated equally for all vehicle types and for all roads, vehicles, and / or environmental conditions. However, in other embodiments, various other factors or variables may also be considered when generating the target trajectory. Different target trajectories can be generated for different types of vehicles (e.g., cars, light trucks, and trailers). For example, a target trajectory with a relatively narrow turning radius can be generated for a small car compared to a larger semi-trailer truck. In some embodiments, road, vehicle, and environmental conditions may also be considered. For example, different target trajectories can be generated for different road conditions (e.g., wet, snowy, icy, dry, etc.), vehicle conditions (e.g., tire condition or estimated tire condition, braking condition or estimated braking condition, remaining fuel, etc.), or environmental factors (e.g., time of day, visibility, weather, etc.). The target trajectory may also depend on one or more aspects or characteristics of a particular road segment (e.g., speed limits, frequency and size of turns, gradient, etc.). In some embodiments, various user settings may also be used to determine the target trajectory, such as set driving modes (e.g., desired driving aggressiveness, economy mode, etc.).

[0325] The sparse map may also include drawn lane markers 2470 and 2480 representing lane markings along road segments. The drawn lane markers may be represented by multiple location identifiers 2471 and 2481. As described above, the location identifiers may include the real-world coordinates of the points associated with the detected lane markers. Similar to the target trajectory in the model, lane markers may also include elevation data and may be represented as curves in three-dimensional space. For example, the curve may be a spline connecting three-dimensional polynomials of appropriate order, and the curve may be computed based on the location identifiers. The drawn lane markers may also include other information or metadata about the lane markers, such as identifiers of the type of lane marker (e.g., between two lanes with the same direction of travel, between two lanes with opposite directions of travel, the edge of the road, etc.) and / or other characteristics of the lane markers (e.g., continuous line, dashed line, single line, double line, yellow, white, etc.). In some embodiments, the drawn lane markers may be continuously updated within the model, for example, using crowdsourcing techniques. The same vehicle can upload location identifiers during multiple trips on the same road segment, or data can be selected from multiple vehicles (such as 1205, 1210, 1215, 1220, and 1225) traveling on the road segment at different times. The sparse map 800 can then be updated or refined based on subsequent location identifiers received from the vehicles and stored in the system. When updating and refining the drawn lane markings, the updated road navigation model and / or sparse map can be distributed to multiple autonomous vehicles.

[0326] Generating drawn lane markings in sparse maps may also include detecting and / or mitigating errors based on anomalies in the image or the actual lane markings themselves. Figure 24F An exemplary anomaly 2495 associated with the detection of lane marking 2490 is shown. Anomaly 2495 may appear in an image captured by vehicle 200, for example, from an object obstructing the camera's view of the lane marking, debris on the lens, etc. In some instances, the anomaly may be due to the lane marking itself, which may be damaged or worn, or partially covered, for example, by dust, debris, water, snow, or other materials on the road. Anomaly 2495 may result in the vehicle 200 detecting an error point 2491. Sparse map 800 can provide corrections to the drawn lane markings and eliminate errors. In some embodiments, vehicle 200 may detect error point 2491, for example, by detecting anomaly 2495 in an image or by identifying errors based on lane marking points detected before and after the anomaly. Based on the detection of the anomaly, the vehicle may ignore point 2491 or adjust it to be consistent with other detected points. In other embodiments, the error may be corrected after the point has been uploaded, for example, by determining that the point is outside an expected threshold based on other points uploaded during the same trip or based on aggregation of data from previous trips along the same road segment.

[0327] Lane markings drawn in the navigation model and / or sparse map can also be used for navigation of autonomous vehicles traversing the corresponding roads. For example, a vehicle navigating along a target trajectory can periodically use lane markings drawn in the sparse map to align itself with the target trajectory. As mentioned above, between landmarks, vehicles can navigate based on dead reckoning, using sensors to determine their own motion and estimate their position relative to the target trajectory. Over time, errors may accumulate, and the vehicle's position determination relative to the target trajectory may become increasingly inaccurate. Accordingly, vehicles can use lane markings appearing in the sparse map 800 (and their known locations) to reduce errors in position determination caused by dead reckoning. In this way, the lane markings identified in the sparse map 800 can serve as navigation anchors by which the vehicle's accurate position relative to the target trajectory can be determined.

[0328] Figure 25A An exemplary image 2500 of the environment surrounding a vehicle is shown, which can be used for navigation based on drawn lane markings. For example, image 2500 can be captured by vehicle 200 via image capture devices 122 and 124 included in image acquisition unit 120. Image 2500 may include an image of at least one lane marking 2510, such as... Figure 25AAs shown. Image 2500 may also include one or more landmarks 2521, such as road signs, for navigation as described above. Figure 25A Some of the elements shown are also shown for reference, such as elements 2511, 2530 and 2520, which do not appear in the captured image 2500 but are detected and / or identified by the vehicle 200.

[0329] Using the above description of... Figures 24A-24D and Figure 24F Using various techniques, vehicles can analyze image 2500 to identify lane markings 2510. Individual points 2511 corresponding to features of the lane markings in the image can be detected. For example, point 2511 may correspond to the edge of a lane marking, a corner of a lane marking, the midpoint of a lane marking, a vertex between two intersecting lane markings, or various other features or locations. Point 2511 can be detected as corresponding to the location of points stored in a navigation model received from a server. For example, if a sparse map containing points representing the centerline of drawn lane markings is received, point 2511 can also be detected based on the centerline of lane marking 2510.

[0330] The vehicle can also determine its longitudinal position, represented by element 2520 and located along the target trajectory. The longitudinal position 2520 can be determined from image 2500, for example, by detecting landmarks 2521 within image 2500 and comparing the measured position with known landmark positions stored in the road model or sparse map 800. The vehicle's position along the target trajectory can then be determined based on the distance to the landmark and the known location of the landmark. The longitudinal position 2520 can also be determined from images other than those used to determine the position of lane markings. For example, the longitudinal position 2520 can be determined by detecting landmarks in images from other cameras within image acquisition unit 120 that were captured simultaneously or nearly simultaneously with image 2500. In some instances, the vehicle may not be near any landmark or other reference point used to determine the longitudinal position 2520. In such instances, the vehicle can navigate based on dead reckoning and can therefore use sensors to determine its own motion and estimate its longitudinal position 2520 relative to the target trajectory. The vehicle can also determine distance 2530, which represents the actual distance between the vehicle and lane markings 2510 observed in the captured images(s). When determining distance 2530, camera angle, vehicle speed, vehicle width, or various other factors may be considered.

[0331] Figure 25BLateral positioning correction of a vehicle based on lane markings drawn in a road navigation model is illustrated. As described above, vehicle 200 can determine the distance 2530 between vehicle 200 and lane markings 2510 using one or more images captured by vehicle 200. Vehicle 200 may also have access to a road navigation model, such as a sparse map 800, which may include drawn lane markings 2550 and a target trajectory 2555. The drawn lane markings 2550 can be modeled using the techniques described above, such as using crowdsourced location identifiers captured by multiple vehicles. The target trajectory 2555 can also be generated using various techniques described previously. As described above regarding... Figure 25A As described, the vehicle 200 can also determine or estimate its longitudinal position 2520 along the target trajectory 2555. The vehicle 200 can then determine a projected distance 2540 based on the lateral distance between the target trajectory 2555 and a drawn lane marking 2550 corresponding to the longitudinal position 2520. The lateral positioning of the vehicle 200 can be corrected or adjusted by comparing the actual distance 2530 measured using captured images(s) with the projected distance 2540 from the model.

[0332] Figure 26A This illustrates an exemplary process 2600A for drawing lane markings for use in autonomous vehicle navigation, conforming to a disclosed embodiment. In step 2610, process 2600A may include receiving two or more location identifiers associated with detected lane markings. For example, step 2610 may be performed by server 1230 or one or more processors associated with the server. (As described above regarding...) Figure 24EAs described, the location identifier may include the location in real-world coordinates of a point associated with a detected lane marking. In some embodiments, the location identifier may also include other data, such as additional information about a road segment or lane marking. Additional data may also be received during step 2610, such as accelerometer data, rate data, landmark data, road geometry or profile data, vehicle positioning data, ego motion data, or various other forms of data described above. The location identifier may be generated by a vehicle (such as vehicles 1205, 1210, 1215, 1220, and 1225) based on images captured by the vehicle. For example, the identifier may be determined based on: acquiring at least one image representing the environment of the main vehicle from a camera associated with the main vehicle; analyzing at least one image to detect lane markings in the environment of the main vehicle; and analyzing the at least one image to determine the location of the detected lane marking relative to the location associated with the main vehicle. As described above, lane markings may include a variety of different marking types, and the location identifier may correspond to various points relative to lane markings. For example, when the detected lane marking is part of a dashed line marking the lane boundary, these points may correspond to the detected corners of the lane marking. When the detected lane marking is part of a continuous line marking the lane boundary, these points may correspond to the detected edges of the lane marking, having various spacings as described above. In some embodiments, such as Figure 24C As shown, these points can correspond to the center line of the detected lane markings, or as... Figure 24D As shown, these points can correspond to the vertex between two intersecting lane markers and at least one or two other points associated with the intersecting lane markers.

[0333] In step 2612, process 2600A may include associating the detected lane marking with a corresponding road segment. For example, server 1230 may analyze real-world coordinates or other information received during step 2610 and compare the coordinates or other information with location information stored in an autonomous vehicle road navigation model. Server 1230 may determine the road segment in the model corresponding to the real-world road segment where the lane marking was detected.

[0334] In step 2614, process 2600A may include updating the autonomous vehicle road navigation model relative to the corresponding road segment based on two or more location identifiers associated with the detected lane markings. For example, the autonomous road navigation model may be a sparse map 800, and server 1230 may update the sparse map to include or adjust the drawn lane markings in the model. Server 1230 may base its updates on the above-mentioned... Figure 24EVarious methods or processes are described to update the model. In some embodiments, updating the autonomous vehicle road navigation model may include storing indicators of one or more locations in the real-world coordinates of detected lane markings. Figure 24E As shown, the autonomous vehicle road navigation model may also include at least one target trajectory for the vehicle to follow along the corresponding road segment.

[0335] In step 2616, process 2600A may include distributing the updated autonomous vehicle road navigation model to multiple autonomous vehicles. For example, server 1230 may distribute the updated autonomous vehicle road navigation model to vehicles 1205, 1210, 1215, 1220, and 1225, which can use the model for navigation. The autonomous vehicle road navigation model may be distributed via one or more networks (e.g., via cellular networks and / or the Internet, etc.) through wireless communication path 1235, such as... Figure 12 As shown.

[0336] In some embodiments, lane markings may use data received from multiple vehicles (e.g., through crowdsourcing technologies, as discussed above). Figure 24E (As described) to draw. For example, process 2600A may include: receiving a first communication from a first primary vehicle, the first communication including a location identifier associated with the detected lane marking; and receiving a second communication from a second primary vehicle, the second communication including an additional location identifier associated with the detected lane marking. For example, the second communication may be received from a subsequent vehicle traveling on the same road segment, or from the same vehicle on a subsequent journey along the same road segment. Process 2600A may further include: refining the determination of at least one location associated with the detected lane marking based on the location identifier received in the first communication and based on the additional location identifier received in the second communication. This may include: using an average of multiple location identifiers, and / or filtering out “ghost” identifiers that may not reflect the real-world location of the lane marking.

[0337] Figure 26BThis is a flowchart illustrating an exemplary process 2600B for autonomously navigating a primary vehicle along a road segment using drawn lane markings. Process 2600B can be executed, for example, by the processing unit 110 of the autonomous vehicle 200. In step 2620, process 2600B may include receiving an autonomous vehicle road navigation model from a server-based system. In some embodiments, the autonomous vehicle road navigation model may include a target trajectory for the primary vehicle along the road segment and location identifiers associated with one or more lane markings associated with the road segment. For example, vehicle 200 may receive a sparse map 800 or another road navigation model formed using process 2600A. In some embodiments, the target trajectory may be represented as a three-dimensional spline, for example, as... Figure 9B As shown above regarding... Figures 24A-24F As described, location identifiers may include the real-world coordinates of points associated with lane markings (e.g., corner points of dashed lane markings, edge points of consecutive lane markings, vertices between two intersecting lane markings and other points associated with intersecting lane markings, centerlines associated with lane markings, etc.).

[0338] In step 2621, process 2600B may include receiving at least one image representing the environment of a vehicle. This image may be received from an image capturing device of the vehicle, such as image capturing devices 122 and 124 included in image acquisition unit 120. The image may include images of one or more lane markings, similar to image 2500 described above.

[0339] In step 2622, process 2600B may include: determining the longitudinal position of the main vehicle along the target trajectory. (As mentioned above regarding...) Figure 25A As described, this can be based on other information in the captured image (e.g., landmarks, etc.) or by dead reckoning of vehicles between detected landmarks.

[0340] In step 2623, process 2600B may include: determining the expected lateral distance to a lane marker based on the determined longitudinal position of the main vehicle along the target trajectory and based on two or more location identifiers associated with at least one lane marker. For example, vehicle 200 may use sparse map 800 to determine the expected lateral distance to the lane marker. Figure 25B As shown, the longitudinal position 2520 along the target trajectory 2555 can be determined in step 2622. Using the sparse map 800, the vehicle 200 can determine the expected distance 2540 to the drawn lane markings 2550 corresponding to the longitudinal position 2520.

[0341] In step 2624, process 2600B may include: analyzing at least one image to identify at least one lane marking. For example, as described above, vehicle 200 may use various image recognition techniques or algorithms to identify lane markings within the image. For example, such as... Figure 25A As shown, lane marking 2510 can be detected through image analysis of image 2500.

[0342] In step 2625, process 2600B may include: determining the actual lateral distance to at least one lane marking based on analysis of at least one image. For example, a vehicle may determine a distance 2530 representing the actual distance between the vehicle and lane marking 2510, such as... Figure 25A As shown. When determining the distance 2530, factors such as camera angle, vehicle speed, vehicle width, camera position relative to the vehicle, and various other factors can be considered.

[0343] In step 2626, process 2600B may include: determining an autonomous maneuvering action of the primary vehicle based on the difference between the expected lateral distance to at least one lane mark and the determined actual lateral distance to at least one lane mark. For example, as described above regarding Figure 25B As described, vehicle 200 can compare actual distance 2530 with expected distance 2540. The difference between the actual distance and the expected distance indicates the error (and its magnitude) between the vehicle's actual position and the target trajectory to be followed by the vehicle. Accordingly, the vehicle can determine autonomous maneuvering actions or other autonomous actions based on the difference. For example, if the actual distance 2530 is less than the expected distance 2540, such as... Figure 25B As shown, the vehicle can determine autonomous maneuvering actions to guide it away from lane marking 2510. Therefore, the vehicle's position relative to the target trajectory can be corrected. For example, process 2600B can be used to improve vehicle navigation between landmarks.

[0344] Identifying Potential Communication Obstacles

[0345] Figure 27This is a schematic diagram of a system 2700 for determining a route using signal information, conforming to the disclosed embodiments. In some embodiments, the system 2700 and / or devices discussed herein may perform all or part of the processes discussed herein (e.g., process 3000, process 3001). The route may be determined by a single vehicle (e.g., vehicle 2702) or by multiple vehicles (e.g., multiple vehicles 2702). Vehicle 2702 may operate autonomously or semi-autonomously and / or include a navigation system using various different devices. In some embodiments, the vehicle may include a communication system for transmitting, receiving, and / or processing communications received at or sent from the vehicle. The communication system may be a device or system that is part of the vehicle itself, a device or system connected to another vehicle system (e.g., a navigation system), or a device (e.g., a smartphone) that is separate from and non-communicatively connected to the vehicle. In some examples, the communication system may be associated with functions related to the vehicle (e.g., a navigation system, a teleoperation system), and / or may be connected to services associated with the vehicle (e.g., a multimedia provisioning service associated with the vehicle and / or operating as part of a robotaxi service). For example, the navigation system in vehicle 2702 may be connected to the communication system, which may include antennas, networking controllers, GPS devices, cellular transceivers and / or Wi-Fi transceivers, and so on. The navigation system may use information provided by one or more of those devices (or in combination with other information provided by other devices, such as one or more cameras capturing images of the environment of vehicle 2702) to navigate. Furthermore, the communication system may use GPS devices, cellular transceivers and / or Wi-Fi transceivers, via one or more networks, to receive navigation information and transmit the navigation information to one or more remote servers (e.g., server 1230). Navigation information may include captured images or other information about the environment of vehicle 2702 (e.g., road experience management or REM, maps, image frames, video content (including live streams), depth information (point clouds), telemetry data, object classification and location, error reports, usage / operation statistics and / or various other types of data collected by the vehicle's onboard sensors).

[0346] Additionally, the navigation system can determine the current local position of vehicle 2702 relative to one or more maps, with vehicle 2720 having access to one or more maps. In some cases, the maps may include REM maps. Such REM maps may include sparse maps that store a representation (e.g., a 3D spline representation) of the target trajectory of the main vehicle (e.g., vehicle 2702) along a road segment and / or along a navigable road lane (or any other drawn surface, such as a parking lot, etc.) of the road segment, represented by the REM map. The REM map may also store landmark identifiers and the associated refined locations of those landmarks. Lane identification, landmark identification, landmark location determination, object identification, and object location determination can all be determined by collecting such information during multiple drives of multiple vehicles along navigable roads / areas. The collected information (e.g., crowdsourced information) may be integrated and / or aligned (e.g., via a remote server) to determine refined target trajectories, refined landmark locations, etc., for storage in the REM map. Subsequently, the primary vehicle navigating relative to the REM map can operate, for example, by the following steps: capturing an image of the primary vehicle's environment; identifying potential landmarks in the captured image; confirming landmark identifiers based on information stored in the REM map; determining landmark locations based on the REM map; and then using those determined landmark locations, determining the localized location of the primary vehicle along the target trajectory using (multiple) REM maps. In other examples, the map may include an HD map and / or a REM map to which additional information layers (such as business information, infotainment services, etc.) have been added.

[0347] The primary vehicle (e.g., vehicle 2702) can receive map information (e.g., REM maps) via a wireless connection (e.g., cellular, Wi-Fi, etc.) between the primary vehicle and a server (e.g., server 1230). Additionally, the primary vehicle can use GPS for vehicle positioning relative to the map information. When such a wireless connection is interrupted, unavailable, or degraded, it may result in interruptions in the operation of the autonomous vehicle or a loss of autonomous driving capabilities. Other operations may also be affected, such as non-autonomous navigation communications, multimedia communications, and other communications, as discussed below. In a further example, the primary vehicle (e.g., vehicle 2702) may be part of a data collection or harvesting service and may be configured to transmit data collected by the primary vehicle via a wireless connection (e.g., cellular, Wi-Fi, etc.) between the primary vehicle and a server (e.g., server 1230). Examples of such data may include image frames, video content (including live streams), depth information (point clouds, depth maps, etc.), telemetry data, object classification and location, and / or various other types of data collected by onboard sensors on vehicles and / or from nearby data sources (e.g., smart road infrastructure).

[0348] For example, the quality characteristics of signals received by a vehicle system (e.g., navigation system, communication system, etc.) may vary for various reasons. In some embodiments, differences in signal quality may be attributed to geographical conditions (e.g., natural or man-made objects obstructing direct line of sight from the signal source to the vehicle) or a lack of coverage for transceivers that can broadcast signals to and receive transmissions from the navigation system (e.g., a lack of cell towers or Wi-Fi hotspots in certain areas (e.g., rural areas), while urban environments may include Wi-Fi hotspots near roads, more cell towers, dense user concentration in relatively small areas, etc.). Additionally, other unwanted signals, such as electromagnetic (EMI) emissions within the area, may affect the operation of the navigation system. Accordingly, it would be beneficial for vehicles to be able to consider and, where necessary, avoid (or modify navigation planning to consider) the following areas: areas that may have reduced signal quality, areas with no signal sources at all, areas with EMI emissions that may interfere with the vehicle system, or areas otherwise not associated with one or more communication signal characteristics consistent with the target.

[0349] In some embodiments, vehicle 2702 may include a navigation system (e.g., an autonomous driving navigation system) capable of accessing map information (e.g., a communication system using the vehicle). The map information can provide location information relating to roads and / or landmarks (e.g., signs, buildings, traffic lights, etc.) along a route or within a geographic area. The map information may be stored locally (e.g., in a database onboard vehicle 2702) and / or remotely (e.g., in a database stored on a remote server accessible via a network). The navigation system may further use one or more cameras onboard vehicle 2702 to determine its location with reference to map information (e.g., REM maps discussed above). For example, vehicle 2702 may include one or more cameras and may determine the vehicle's location based on image analysis with reference to map information. In some implementations, the mapping system may include Road Experience Management (REM) technology. The navigation system may further be attached to or replace images captured by the imagery system to use one or more devices, such as GPS devices, cellular transceivers, and / or Wi-Fi transceivers (as further discussed herein). As discussed earlier, the performance of a vehicle system (e.g., a navigation system) may be adversely affected in areas where these devices cannot successfully receive or transmit, or where there are areas where EMI emissions that could interfere with the navigation system's electronic equipment and / or other vehicle systems may occur.

[0350] In some embodiments, vehicle 2702 may be navigating (e.g., traveling along a determined route) or preparing to navigate (e.g., determining an initial route) to destination 2708 (e.g., street address, coordinate location, point within a defined area, etc.). For example, vehicle 2702 and / or a remote server may determine navigation information, map information, signal information, etc., available for vehicle 2702 to navigate to destination 2708. In some embodiments, vehicle 2702 may navigate along road network 2704, which may allow multiple potential routes to destination 2708, such as routes 2706a and 2706b. In some embodiments, as mentioned above, a route may coincide with or fall within a predetermined distance of at least one location associated with reduced, non-existent, or otherwise undesirable quality characteristics (discussed further below). For example, route 2706a may pass through location 2710, which may be associated with reduced quality characteristics (e.g., reduced signal strength). In some embodiments, the device (e.g., server 1230) may determine that a location (e.g., location 2710) is associated with specific signal information (e.g., according to processes 3000 and / or 3001). Based on this determination, a route 2706b may be determined (e.g., according to process 3000), which may differ from route 2706a. In some embodiments, route 2706b may be more desirable than route 2706a. For example, route 2706b may avoid location 2710, may avoid location 2710 more than route 2706a, may have at least one more desirable characteristic than route 2706a, or may have a more optimized combination of characteristics than route 2706a (e.g., as per [reference to...]). Figure 30 (as discussed), etc. In some embodiments, route 2706a may be a modification of route 2706a. For example, route 2706a may partially overlap with route 2706b. In other embodiments, route 2706a may not overlap with route 2706b at all. In some embodiments, the modification may be determined locally or at a remote source (e.g., server 1230) by vehicle 2702 (e.g., according to process 3000 and / or process 3100). In some embodiments, the modified route may be sent to vehicle 2702 and / or determined by vehicle 2702, and vehicle 2702 may change the current route (e.g., route 2706a) to the modified route (e.g., route 2706b). Such modifications may include: changing data at vehicle 2702 (e.g., changing current navigation data, changing information displayed on the vehicle's display, changing the vehicle's autonomous operation, changing the estimated arrival time, alerting the driver, alerting entities associated with the destination, etc.).

[0351] Figure 28 This is a schematic diagram of a system 2800 for determining routes between multiple vehicles using signal information, conforming to the disclosed embodiments. In some embodiments, the system 2800 and / or devices discussed herein may perform all or part of the processes discussed herein (e.g., processes 3000, 3001). In some embodiments, conforming to the disclosed embodiments, vehicle 2802a may be navigating and / or preparing to navigate to a destination, such as destination 2806. For example, vehicle 2802a and / or a remote server (e.g., server 1230) may determine route 2808a, along which vehicle 2802a may travel to reach destination 2806. In some embodiments, vehicle 2802a may navigate along a road network 2804, which may allow multiple potential routes to destination 2806, such as route 2808a. In some embodiments, destination 2806 may be associated with a specific task that may be performed by any one of the multiple vehicles (e.g., vehicle 2802a, vehicle 2802b, etc.). For example, destination 2806 can be associated with picking up passengers (e.g., taxi or ride-sharing services), product delivery, service delivery, etc.

[0352] In some embodiments, route 2808a may pass through or be near location 2810, which may be associated with reduced quality characteristics (e.g., reduced signal strength) or preferred quality characteristics (e.g., determined based on one or more parameters, as discussed below). In some embodiments, a device (e.g., server 1230) may determine that a location (e.g., location 2810) is associated with specific signal information (e.g., according to process 3000 or 3100). Based on this determination, the device may determine that another vehicle and / or route is more suitable for being guided to destination 2806. For example, a remote server may determine that vehicle 2802b can also reach destination 2806 and / or complete the task associated with the destination, and may determine that vehicle 2802b can follow route 2808b to reach destination 2806. As a further example, the remote server and / or vehicle may determine (e.g., according to process 3000 or 3001) that route 2808b is more desirable than route 2808a. For example, route 2808b may avoid location 2810, may avoid location 2810 more thoroughly than route 2808a, may have at least one more desirable characteristic than route 2808a, and may have a more optimized combination of characteristics than route 2808a (such as regarding...). Figure 30 , Figure 31(as discussed elsewhere), etc. In some embodiments, route 2808a may partially overlap with route 2808b. In other embodiments, route 2808a may not overlap with route 2808b. In some embodiments, vehicle 2808b may be guided to destination 2806 according to the route, and / or vehicle 2808a may be instructed not to proceed to destination 2806 (e.g., its route may be cancelled). Regarding examples Figure 31 Further examples of these aspects will be discussed.

[0353] Figure 29 This is a schematic diagram of a system 2900 for generating navigation information using signal information from vehicles, conforming to the disclosed embodiments. In some embodiments, the system 2900 and / or devices discussed herein may perform all or part of the processes discussed herein (e.g., processes 3000, 3001). For example, a fleet of vehicles, such as vehicles 2910, 2912, and 2914, may collect information from GPS, cellular, or Wi-Fi sources or areas with EMI emissions, relating this information to areas with high-quality signals (e.g., signals exceeding a predetermined threshold margin), sufficient signals (e.g., signals meeting a predetermined threshold), degraded signals, or no signals. For example, vehicles may determine signal-related information based on signals detected and / or received from network devices 2904 (e.g., cellular network towers, Wi-Fi hotspots, and / or any other communication networking devices). Then, the vehicle (e.g., vehicle 2910) can transmit (e.g., provide a report) such information identifying these locations to a remote system (e.g., system 2902), which can be used to update information (e.g., signal information, map information, navigation information) and can be used by the vehicle (e.g., vehicles of a convoy).

[0354] In some embodiments, consistent with the disclosed embodiments, system 2902 may include a server, multiple servers, and / or other suitable devices for updating information for vehicles. In some embodiments, signal information or other information for vehicles may be transmitted to system 2902 from provider 2920 (e.g., a separate device and / or system detached from system 2902). For example, provider 2920 may be a system associated with a network communication provider (e.g., a cellular network host, satellite host, etc.), which may be associated with network device 2904. In this way, system 2902 may receive information from vehicles to which it is connected, but may also receive additional information (e.g., signal information) from other vehicles (e.g., vehicle 2914) to which system 2902 may not be connected, but which may send information (e.g., signal information) to another source (e.g., provider 2920). Accordingly, information may be continuously crowdsourced and taken into consideration. For example, the disclosed embodiments may consider changes in signal quality characteristics not only based on geography but also based on time (e.g., an area without a cellular tower may have a cellular tower installed and is therefore no longer designated as an area lacking cellular service, while an area may experience network outages and may be designated as lacking cellular service).

[0355] In some embodiments, the vehicle may determine signal information that can supplement map information used by a navigation system with information identifying one or more locations, where the signal sources discussed above (e.g., GPS, cellular, satellite, and / or Wi-Fi networks) have at least one specific quality characteristic (e.g., signal strength, bandwidth, latency, reduced signal quality, no signal, etc.) at those locations. As discussed above, a sparse map (e.g., a REM map) may be generated during vehicle navigation. For example, the primary vehicle (e.g., vehicle 2912) may collect information from vehicle driving and aggregate that information to populate a map (e.g., a REM map). Such information may be collected from a fleet of vehicles equipped with such navigation systems (e.g., vehicles 2910, 2912, and 2914), which may then upload vehicle driving information (e.g., including signal information, image information, sensor information, calculations, etc.) to a server. In some cases, as further discussed below, aggregated information provided by a fleet of vehicles may include indicators of: GPS location data, GPS signal strength and / or accuracy, Wi-Fi strength and / or availability, cellular strength and / or availability, location associated with EMI emissions, and so on. Because map and / or signal information can identify locations, routes, or areas where signal quality is below standard or nonexistent (e.g., based on signal information), navigation systems can select routes that avoid such areas or reduce the time spent traversing them.

[0356] Figure 30 This is a flowchart illustrating an exemplary process 3000 for determining route information for a vehicle, conforming to the disclosed embodiments. Process 3000 may be executed by a processor of a processing device, such as a processor for a navigation system (e.g., a navigation system with a vehicle), a remote server (e.g., server 1230), multiple servers, a remote system (e.g., system 2902), or any combination thereof. Any and all steps in process 3000 are optional and may be removed, rearranged, and / or reordered in any combination.

[0357] In some embodiments, process 3000 may perform step 3010. In step 3010, process 3000 may obtain the location of the vehicle. In some embodiments, obtaining the location of the vehicle may include determining the location based on GPS data or other location data associated with the vehicle. Such data may be generated by the vehicle (e.g., vehicle 2702), a remote server, a satellite, and / or other computing devices, and / or may be received at the vehicle (e.g., vehicle 2702), the remote server, the satellite, and / or other computing devices. For example, a remote server may determine the location of the vehicle based on location data received from the vehicle, which may in turn have been determined through communication between the vehicle, which is connected to a communication network, and at least one device connected to the communication network (e.g., cellular, satellite, other vehicles, Wi-Fi hotspot, route device, etc.). In some embodiments, the location of the vehicle may be determined using at least one image captured by a camera, which may be included in or on the vehicle.

[0358] In some embodiments, process 3000 may perform step 3020. In step 3020, process 300 may access the map information discussed above. In some embodiments, the accessed map information may be stored at the vehicle (e.g., in the memory component of the vehicle's navigation system), at a remote source (e.g., a remote server), and / or in a combination of these devices. In some embodiments, the remote source may transmit the map information to the vehicle, which may be used by the vehicle's navigation system (e.g., to autonomously drive the vehicle to its destination).

[0359] In step 3030, process 300 may determine a route, which may be a route from the location of a vehicle (e.g., vehicle 2702) to a destination (e.g., destination 2708). In some embodiments, the route may be determined by the vehicle for its planned route. Additionally or alternatively, the route (e.g., the route from the location of the vehicle to the destination) may be determined by a remote source (e.g., a remote server, such as server 1230) and / or may be obtained from a remote source (e.g., by vehicle 2702). In some embodiments, the route may be determined based on map information received from at least one remote server. In some embodiments, the vehicle may be configured for autonomous or semi-autonomous driving.

[0360] In step 3040, process 3000 may receive signal information indicating at least one aspect of the quality characteristics and / or availability of a signal associated with at least one location along the route. In some embodiments, the signal information may indicate multiple quality characteristics. The signal may include satellite signals, cellular signals, Wi-Fi signals, radio signals, and / or any data transmission signals. Quality characteristics may include those of the signal that may affect the operation of equipment associated with the vehicle, the driver of the vehicle, the passengers of the vehicle, and / or tasks associated with the vehicle (e.g., passenger pick-up). For example, quality characteristics may include at least one of the following: signal strength, bandwidth, frequency, amplitude, modulation, signal format, communication standard, upload rate, download rate, signal quality (e.g., packet loss rate), network type (4G, 5G, etc.), roaming or native network characteristics, network security level, mobile QoS level, or latency. In some embodiments, the quality characteristics of the signal may be affected by electromagnetic emissions associated with the location (e.g., detected at or near the location, within a predetermined radius of the location). In some embodiments, electromagnetic emissions may be identified as potential sources of signal interference (e.g., overhead power lines).

[0361] In some embodiments, signal information may include information indicating the quality characteristics of a signal within a predetermined radius of a location (e.g., the location of a vehicle, its location along a route, its location along a road, etc.). In some embodiments, signal information may be based on information collected from multiple vehicles (e.g., information about...). Figure 29 (As discussed). Alternatively or additionally, signal information may be based on information from at least one network provider (such as provider 2920 (e.g., regarding...)). Figure 29 The information discussed)

[0362] In some embodiments, the signal information (e.g., the signal information received at step 3040) may be first signal information, and process 3000 may update the first signal information based on second signal information received by a vehicle (e.g., the vehicle for which a route is determined at step 3030). In some embodiments, the second signal information may be received from another vehicle (e.g., a vehicle other than the vehicle for which a route is determined at step 3030).

[0363] In step 3050, process 3000 may determine quality parameters, which may be related to a communication system (e.g., a system of a vehicle associated with transmitting, receiving, and / or processing communications to and / or from the vehicle, such as regarding...). Figure 27 The quality parameter of at least one operational characteristic associated with (as discussed). In some embodiments, the communication system may be associated with the navigation system (e.g., as discussed regarding...). Figure 12This is discussed, and is associated with sensors and / or another device or system associated with the vehicle. In some embodiments, the navigation system may be configured to receive map information via a communication system. In some embodiments, the communication system may include mobile devices within the vehicle. For example, the communication system may include a user's mobile device, such as, but not limited to, smartphones, tablets, smartwatches, virtual and / or augmented reality wearable devices, portable gaming devices, and / or multimedia devices (e.g., DVD or Blu-ray players, televisions, music players, radios, satellite radios, etc.).

[0364] Operational characteristics may include any aspect related to the performance of operations performed by the vehicle system and / or connected devices, including but not limited to: vehicle driving operations (e.g., acceleration, braking, steering) or data operations (e.g., downloading navigation and / or signal information to the vehicle, uploading navigation and / or signal information from the vehicle, downloading data to user equipment within the vehicle, and uploading data from user equipment within the vehicle). As a further example, operational characteristics may include: receiving map information for navigating the vehicle, receiving video or audio information (e.g., navigation instructions, telephone call data, video call data, etc.), and / or receiving teleoperation communication information or multimedia information.

[0365] Quality parameters can be limitations, ranges, algorithms, values, variables, or other representations used (e.g., using signal information) to determine the suitability of a route. In some embodiments, quality parameters may be based on at least one of the following: predetermined system parameters (e.g., parameters defined by the vehicle system, parameters defined by a remote management source, etc.), user preferences (e.g., parameters set by a user, such as at a user interface within the vehicle), or network activity (e.g., network communication between the vehicle system and another networked device). In some embodiments, network activity may include devices using wireless network connections within the vehicle (e.g., mobile devices as discussed above). As an example, quality parameters may include at least one of the following: signal strength, bandwidth, frequency, amplitude, modulation, signal format, communication standard, upload rate, download speed, signal quality (e.g., packet loss rate), network type (4G, 5G, etc.), roaming or native network characteristics, network security level, mobile QoS level, or latency.

[0366] In some embodiments, quality parameters may include measurements of at least one of the following: signal strength, bandwidth, frequency, amplitude, modulation, signal format, upload rate, download rate, signal quality (e.g., packet loss rate), network type (4G, 5G, etc.), roaming or native network characteristics, network security level, mobile QoS level, or latency. In some embodiments, the measurement may be a statistical measurement based on data from one or more vehicles. For example, the measurement may be a median, mean, range, percentile, weighted value, time-dependent value, and / or context-dependent value (e.g., depending on the operating context of the vehicle, such as speed, traffic light location, weather conditions, etc.). As a further example, the measurement may be determined based on multiple values ​​(e.g., signal strength values), which may be associated with the same or similar locations, and / or may be available from a single vehicle, multiple vehicles, and / or network devices. As new data (e.g., from a fleet of vehicles) is collected, the measurement may be dynamically updated (e.g., by system 2902). In some embodiments, the quality parameters may include a profile that may vary in time or space, such that different parameters and / or weights exist for the signal at certain times and / or locations / regions (e.g., for comparison with signal information).

[0367] In some embodiments, a vehicle (e.g., the vehicle for which a route is determined at step 3030) and / or a remote server (e.g., server 1230) may determine whether signal information (e.g., signal information received at step 3040) meets quality parameters (e.g., quality parameters determined at step 3050). In some embodiments, the signal information may not meet the quality parameters, which may affect the route used for the vehicle (as discussed below).

[0368] In step 3060, process 3000 may determine at least one modification to the route for the vehicle, which may be based on signal information (e.g., signal information received at step 3040) and quality parameters (e.g., quality parameters determined at step 3050). For example, at least one modification may be determined based on a comparison of the route parameters with a threshold. As another example, at least one modification may be determined based on a comparison of parameters (e.g., quality parameters, route parameters discussed further below, thresholds, etc.) with values ​​and / or combinations of values ​​associated with the modified route. As an example, a threshold for a quality parameter (e.g., a threshold for signal strength, bandwidth, etc.) may be compared with at least one route characteristic (e.g., a value associated with the modified route). For example, the signal information may be evaluated against a predetermined threshold. As a further example, the predetermined threshold may represent a quality level (e.g., a level of quality characteristics) previously determined to provide sufficient signal strength for one or more predetermined tasks (e.g., transmitting a video stream to a mobile device communicatively connected to the vehicle). The navigation system can select alternative or modified routes to navigate within areas where sufficient signal strength is provided from one or more of these sources (e.g., GPS sources, cellular sources, and / or Wi-Fi sources). For example, the navigation system can select routes with better GPS, cellular, or Wi-Fi signals, or routes with more cellular towers or more Wi-Fi hotspots. Similarly, since signal information can identify areas where electromagnetic (EMI) emissions might affect the operation of electronic devices in the navigation and / or communication systems, the navigation system can avoid such areas and select alternative routes. Alternatively or additionally, process 3000 can determine at least one modification to the route for the vehicle based on multiple quality parameters.

[0369] In some embodiments, multiple modified routes may be determined, and one of the multiple routes may be selected, for example, based on quality parameters, route parameters, and / or user selection. For example, process 3000 may search for the best match (e.g., a combination that satisfies quality and / or route parameters), or select from matched routes or the best route (or points that can be connected by available paths to provide the route). In some embodiments, the modification may be determined based on map information received from at least one remote server.

[0370] In some embodiments, modifications may be determined based on route parameters (e.g., parameters that differ from quality parameters associated with the quality characteristics of the same signal information). For example, the signal information received at step 3040 may be first signal information, and the route parameters may include at least one of the following: second signal information indicating a second quality characteristic of the signal associated with at least a second location along the second route, travel time, distance, energy consumption, number of potential additional passengers, navigation preferences (e.g., preferences to avoid left turns, preferences to avoid right turns, preferences to avoid toll roads, preferences to avoid busy intersections, preferences to avoid highways, preferences to avoid traffic, etc.), safety preferences (e.g., preferences to avoid high-accident areas), network type preferences (e.g., 4G networks, 5G networks, Wi-Fi networks, satellite constellations, etc.), or network provider preferences (e.g., preferences to prioritize a particular network provider relative to other network providers). In some embodiments, route parameters may be compared with route characteristics (e.g., expected values ​​associated with the determined route, such as expected travel time). In some embodiments, modifications may be determined based on such comparisons. Alternatively or additionally, process 3000 may determine at least one modification to the route for a vehicle based on multiple route parameters.

[0371] In some embodiments, quality parameters and route parameters can be weighted according to a scoring scheme. For example, a computing device (e.g., a processing device with a vehicle, server 1230, etc.) can correlate at least one quality parameter and at least one route parameter with each other, and these parameters can have different weights. In some embodiments, the computing device can optimize the parameters based on their weights. In some embodiments, the quality parameter can depend on the route parameter. For example, the quality parameter can be optimized after the route parameter (e.g., for a modified route) reaches a predetermined threshold. In other embodiments, the route parameter can depend on the quality parameter. For example, the route parameter can be optimized after the quality parameter (e.g., for a modified route) reaches a predetermined threshold.

[0372] Additionally, the navigation system can sort routes or locations and determine whether to drive through such areas. Accordingly, the disclosed systems and methods can use a scoring scheme (e.g., a weighted method) to determine whether to drive a particular route or pass through a particular area. For example, if an area suffers from degraded cellular signal but has Wi-Fi hotspots, there may be no reason to avoid that area. As another example, driving a route may be acceptable if it has sufficient GPS coverage but no cellular towers. In such examples, however, alternative routes with good GPS reception and cellular coverage may be determined to be less preferred overall due to one or more predetermined criteria (e.g., the route may involve longer travel time and / or longer distance, greater risk, driving on toll roads, etc.). In some embodiments, the predetermined criteria may be determined by the user (e.g., input into the navigation system using the vehicle's infotainment center interface).

[0373] Furthermore, process 3000 can also use signal information identifying areas with no signal source (or reduced signal) to determine when to switch networks. For example, if the signal information indicates that an area has no Wi-Fi coverage but has cellular service, the navigation system can switch communication from the Wi-Fi network to the cellular network when driving into that area (as discussed further below). As another example, routes can be selected based on available signal strength (e.g., in route selection, more weight can be assigned to higher signal strength). As yet another example, routes can be selected based on predetermined service providers or bearers (e.g., the network of a preferred cellular provider as discussed above). Accordingly, different service providers or bearers can be associated with different scores (or weights) and selected accordingly. For example, if the signal strength of the network of the preferred bearer along the route meets a predetermined threshold, a route with coverage provided by the preferred bearer can be selected relative to a route with coverage provided by a second bearer, even if the signal strength of the se...

Claims

1. A navigation system comprising: at least one processor comprising circuitry and having access to a memory, wherein the memory comprises instructions that, when executed by the circuitry, cause the at least one processor to: obtain a route from a location of a first vehicle to a destination; receive signal information obtained by a second vehicle, the signal information indicating a quality characteristic of a signal associated with at least one location along the route traveled by the second vehicle; obtain a quality parameter threshold for at least one operational characteristic associated with a communication system, the at least one operational characteristic associated with receiving map information that can be used by the first vehicle for autonomous or semi-autonomous navigation; and determine at least one modification to the route for the first vehicle based on a comparison of the received signal information to the quality parameter threshold.

2. The navigation system of claim 1, wherein, the first vehicle is configured for autonomous or semi-autonomous driving.

3. The navigation system of claim 1, wherein, the memory further comprises instructions that, when executed by the circuitry, cause the at least one processor to obtain the location of the first vehicle.

4. The navigation system of claim 1, wherein, the communication system is associated with the navigation system or the communication system comprises a mobile device within the first vehicle.

5. The navigation system of claim 4, wherein, the navigation system is configured to receive the map information via the communication system.

6. The navigation system of claim 1, wherein, the at least one modification to the route comprises driving around the at least one location along the route.

7. The navigation system of claim 6, wherein, the signal information indicating the quality characteristic of the signal associated with the at least one location along the route does not satisfy the quality parameter threshold.

8. The navigation system of claim 1, wherein, the at least one modification to the route reduces travel time in the vicinity of the at least one location along the route.

9. The navigation system of claim 1, wherein, the at least one modification to the route maintains or increases travel time in the vicinity of the at least one location along the route.

10. The navigation system of claim 1, wherein, the quality parameter threshold is based on at least one of: a predetermined system parameter, a user preference, or network activity.

11. The navigation system of claim 10, wherein, the network activity comprises a device within the first vehicle using a wireless network connection.

12. The navigation system of claim 1, wherein, the quality parameter threshold comprises at least one of: signal strength, bandwidth, latency, or a profile that varies in time or space.

13. The navigation system of claim 12, wherein, the at least one operational characteristic comprises at least one of: receiving video or audio information; or receiving teleoperation communication information or multimedia information.

14. The navigation system of claim 1, wherein: the signal information is first signal information, the route is a first route; and the at least one modification to the route is further determined based on route parameters comprising at least one of: second signal information indicating a second quality characteristic of a signal associated with at least a second location along a second route, travel time, distance, degree of energy consumption, number of potential additional passengers, navigation preference, safety preference, network type preference, or network provider preference.

15. The navigation system of claim 14, wherein, the at least one modification is determined based on a comparison of the route parameters to a threshold.

16. The navigation system of claim 14, wherein, the quality parameter threshold and the route parameters are weighted according to a scoring scheme.

17. The navigation system of claim 16, wherein, The quality parameter threshold depends on the route parameter.

18. The navigation system of claim 16, wherein, The route parameter depends on the quality parameter threshold.

19. The navigation system of claim 1, wherein, The memory further includes instructions that, when executed by the circuitry, cause the processor to determine at least one point along the route at which to handoff communications of the first vehicle from a first network to a second network.

20. The navigation system of claim 1, wherein, The location of the first vehicle is determined using at least one image captured by a camera included in the first vehicle.

21. The navigation system of claim 1, wherein the signals comprise satellite signals, cellular signals, or Wi-Fi signals.

22. The navigation system of claim 1, wherein the route is determined based on map information received from at least one remote server.

23. The navigation system of claim 1, wherein, The signal information is based on at least one of information collected from a plurality of vehicles or information from at least one network provider.

24. The navigation system of claim 1, wherein, The signal information includes information indicative of the quality characteristic of the signals within a predetermined radius of the at least one location.

25. The navigation system of claim 1, wherein, The quality characteristic of the signals is affected by an electromagnetic emission associated with the location.

26. The navigation system of claim 25, wherein the electromagnetic emission is determined to be a potential source of signal interference.

27. The navigation system of claim 1, wherein: The signal information is first signal information; and The at least one processor is further programmed to update the first signal information based on second signal information received by the first vehicle or the second vehicle.

28. The navigation system of claim 27, wherein, The second signal information is received from a third vehicle.

29. The navigation system of claim 1, wherein, The route from the location of the first vehicle to the destination is obtained from a remote source.

30. The navigation system of claim 1, wherein, The memory further includes instructions that, when executed by the circuitry, cause the processor to: analyze a plurality of reports; and determine an optimized route based on the analysis.

31. The navigation system of claim 30, wherein, Determining the optimized route includes determining, for at least one location along the route prior to optimization, a quality characteristic of signals associated with the at least one location along the route prior to determining the optimized route and a quality characteristic of signals associated with at least one location of the optimized route.

32. The navigation system of claim 1, wherein, The at least one processor is further configured to determine whether the signal information satisfies the quality parameter threshold, and the at least one modification to the route is based on the determination of whether the signal information satisfies the quality parameter threshold.

33. The navigation system of claim 1, wherein, The map information includes a three-dimensional polynomial representation of a trajectory of at least one of the first or second vehicles.

34. A non-transitory computer-readable medium comprising instructions executable by at least one processor to perform a method comprising: obtaining a route from a location of a first vehicle to a destination; receiving signal information obtained by a second vehicle, the signal information indicative of a quality characteristic of signals associated with at least one location along the route traveled by the second vehicle; receiving signal information obtained by a second vehicle, the signal information indicating a quality characteristic of a signal associated with at least one location along the route traveled by the second vehicle; and determining at least one modification to the route for the first vehicle based on a comparison of the received signal information to the quality parameter threshold.

35. A method for vehicle navigation, comprising: obtaining a route from a location of a first vehicle to a destination; receiving signal information obtained by a second vehicle, the signal information indicating a quality characteristic of a signal associated with at least one location along the route traveled by the second vehicle; determining a quality parameter threshold for at least one operational characteristic associated with a communication system, the at least one operational characteristic associated with receiving map information that can be used by the first vehicle for autonomous or semi-autonomous navigation; and determining at least one modification to the route for the first vehicle based on a comparison of the received signal information to the quality parameter threshold.

36. A method for route-guiding autonomous vehicles, comprising: receiving a first location of a first autonomous vehicle; receiving a second location of a second autonomous vehicle; receiving a destination, wherein the destination is closer to the first location of the first autonomous vehicle than to the second location of the second autonomous vehicle; determining a first route from the first location to the destination and a second route from the second location to the destination; receiving first signal information indicating a first quality characteristic of a first signal associated with at least one location along the first route; receiving second signal information indicating a second quality characteristic of a second signal associated with at least one location along the second route; receiving a quality parameter for at least one operational characteristic associated with a first communication system of the first autonomous vehicle and a second communication system of the second autonomous vehicle, the at least one operational characteristic associated with receiving map information that can be used by the first autonomous vehicle or the second autonomous vehicle for autonomous or semi-autonomous navigation; determining whether to route-guide the first autonomous vehicle or the second autonomous vehicle to the destination based on the first location of the first autonomous vehicle, the second location of the second autonomous vehicle, and the quality parameter; and transmitting instruction information configured to cause the second autonomous vehicle to travel from the second location to the destination.

37. A system for route-guiding autonomous vehicles, the system comprising: at least one processor programmed to: receive a first location of a first autonomous vehicle; receive a second location of a second autonomous vehicle; receive a destination, wherein the destination is closer to the first location of the first autonomous vehicle than to the second location of the second autonomous vehicle; determine a first route from the first location to the destination and a second route from the second location to the destination; receiving first signal information indicative of a first quality characteristic of a first signal associated with at least one location along the first route; receiving second signal information indicative of a second quality characteristic of a second signal associated with at least one location along the second route; receiving a quality parameter associated with at least one operational characteristic of a first communication system of the first autonomous vehicle and a second communication system of the second autonomous vehicle, the at least one operational characteristic associated with receiving map information that can be used by the first autonomous vehicle or the second autonomous vehicle to navigate autonomously or semi-autonomously; determining whether to route the first autonomous vehicle or the second autonomous vehicle to the destination based on the first location of the first autonomous vehicle, the second location of the second autonomous vehicle, and the quality parameter; and transmitting instruction information configured to cause the second autonomous vehicle to travel from the second location to the destination.

38. The system of claim 37, wherein, Determining whether to route the first autonomous vehicle or the second autonomous vehicle includes determining that the first signal information does not satisfy the quality parameter.

39. The system of claim 37, wherein, The quality parameter is based on at least one of a predetermined system parameter, a user preference, or network activity.

40. The system of claim 37, wherein, The quality parameter includes at least one of a signal strength, a bandwidth, or a latency.

41. The system of claim 40, wherein, The at least one operational characteristic includes receiving video or audio information.

42. The system of claim 37, wherein: Determining whether to route the first autonomous vehicle or the second autonomous vehicle to the destination is further based on a route parameter; and The quality parameter and the route parameter are weighted according to a scoring scheme.

43. The system of claim 42, wherein, The quality parameter is dependent on the route parameter.

44. The system of claim 42, wherein, The route parameter is dependent on the quality parameter.

45. The system of claim 37, wherein the processor is further configured to determine at least one point along a second autonomous vehicle route at which to switch communication of the second autonomous vehicle from a first network to a second network.

46. The system of claim 37, wherein, At least the first route or the second route is determined further based on a route parameter, the route parameter including at least one of a travel time, a distance, a network type preference, or a network provider preference.

47. The system of claim 37, wherein, The first signal or the second signal includes a satellite signal, a cellular signal, or a Wi-Fi signal.

48. The system of claim 47, wherein the first signal information or the second signal information is based on information collected from a plurality of vehicles.

49. The system of claim 37, wherein, The first signal information or the second signal information is based on information from at least one network provider.

50. The system of claim 37, wherein, The first signal information includes information indicative of the first quality characteristic of the first signal within a predetermined radius of the at least one location along the first route.

51. The system of claim 37, wherein, The second signal information includes information indicative of the second quality characteristic of the second signal within a predetermined radius of the at least one location along the second route.

52. The system of claim 37, wherein, The first quality characteristic or the second quality characteristic is affected by electromagnetic emissions.

53. The system of claim 52, wherein, The electromagnetic emissions are determined to be a potential source of signal interference.

54. The system of claim 37, wherein, The at least one processor is further configured to transmit instruction information configured to cause the first autonomous vehicle to not travel from the first location to the destination.

55. The system of claim 37, wherein, The operational characteristic relates to at least one of: a vehicle driving operation or a vehicle data operation.

Images