Systems and methods for optimizing map tile requests for navigation

Abstract

A system for vehicle navigation can include a processor comprising circuitry and a memory. The memory can include instructions that, when executed by the circuitry, cause the processor to receive navigation information associated with the vehicle including an indicator of a location of the vehicle and determine target navigation map segments to retrieve from a map database. The map database can include stored navigation map segments each corresponding to a real-world region. The determination of the target navigation map segments can be based on the indicator of the vehicle location and map segment connectivity information associated with the stored navigation map segments. The instructions can also cause the processor to initiate a download of the target navigation map segments from the map database and cause the vehicle to navigate along a target trajectory contained in one or more of the target navigation map segments downloaded from the map database.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 994,003, filed March 24, 2020, and U.S. Provisional Application No. 63 / 066,564, filed August 17, 2020. Both applications are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure generally pertains to vehicle navigation. Background Technology

[0004] With continuous technological advancements, the goal of fully autonomous vehicles capable of navigating roads is on the horizon. Autonomous vehicles may need to consider a wide variety of factors and make appropriate decisions based on those factors to safely and accurately reach their intended destination. For example, autonomous vehicles may need to process and interpret visual information (e.g., information captured from cameras), information from radar or lidar, and may also use information from other sources (e.g., from GPS devices, speed sensors, accelerometers, suspension sensors, etc.). Simultaneously, to navigate to their destination, autonomous vehicles may also need to identify their position within a specific road (e.g., a specific lane in a multi-lane road), navigate alongside other vehicles, avoid obstacles and pedestrians, observe traffic signals and signs, and move from one road to another at appropriate intersections or junctions. Harnessing and interpreting the vast amounts of information collected by the vehicle as it reaches its destination presents numerous design challenges. The massive amounts of data that autonomous vehicles may need to analyze, access, and / or store (e.g., captured image data, map data, GPS data, sensor data, etc.) pose challenges that may actually limit or even adversely affect autonomous navigation. Furthermore, if autonomous vehicles rely on traditional mapping technology for navigation, the massive amounts of data required to store and update maps will pose a huge challenge. Summary of the Invention

[0005] Embodiments consistent with this disclosure provide systems and methods for vehicle navigation.

[0006] In one embodiment, a system for navigating a vehicle may include at least one processor, which includes circuitry and memory. The memory includes instructions that, when executed by the circuitry, cause the at least one processor to receive navigation information associated with the vehicle, the navigation information including at least an indicator of the vehicle's position. When executed by the circuitry, the instructions may also cause the at least one processor to determine a plurality of target navigation map segments to be retrieved from a map database. The map database may include a plurality of stored navigation map segments, each corresponding to a real-world region. The determination of the plurality of target navigation map segments may be based on the vehicle's position indicator and on map segment connectivity information associated with the plurality of stored navigation map segments. When executed by the circuitry, the instructions may also cause the at least one processor to initiate the download of the plurality of target navigation map segments from the map database. When executed by the circuitry, the instructions may also cause the at least one processor to navigate the vehicle along at least one target trajectory contained in one or more of the plurality of target navigation map segments downloaded from the map database.

[0007] In one embodiment, a non-transitory computer-readable medium may contain instructions that, when executed by at least one processor, cause the at least one processor to perform operations including receiving navigation information associated with a vehicle. The navigation information may include at least an indicator of the vehicle's position. The operation may further include determining a plurality of target navigation map segments to be retrieved from a map database. The map database may include a plurality of stored navigation map segments, each corresponding to a real-world region. The determination of the plurality of target navigation map segments may be based on the vehicle's position indicator and on map segment connectivity information associated with the plurality of stored navigation map segments. The operation may further include initiating the download of the plurality of target navigation map information from the map database. The operation may further include navigating the vehicle along at least one target trajectory contained in one or more of the plurality of target navigation map segments downloaded from the map database.

[0008] In one embodiment, a non-transitory computer-readable medium may contain a digital map for vehicle navigation. The digital map may include multiple navigation map segments, each corresponding to a real-world region. For each of the multiple navigation map segments: the digital map may further include at least one map segment connectivity indicator associated with each boundary shared with adjacent navigation map segments. The at least one map segment connectivity indicator may be stored as a Boolean bit on the computer-readable medium.

[0009] The foregoing general description and the following detailed description are merely exemplary and illustrative, and are not intended to limit the scope of the claims. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings:

[0011] Figure 1 It is an illustrative representation of an exemplary system consistent with the disclosed embodiments.

[0012] Figure 2A It is an illustrative side view representation of an exemplary vehicle that includes a system consistent with the disclosed embodiments.

[0013] Figure 2B yes Figure 2A The illustrated top view of the vehicle and system shown is consistent with the disclosed embodiments.

[0014] Figure 2C This is an illustrative top view representation of another embodiment of a vehicle that includes a system consistent with the disclosed embodiments.

[0015] Figure 2D This is an illustrative top view representation of yet another embodiment of a vehicle that includes a system consistent with the disclosed embodiments.

[0016] Figure 2E This is an illustrative top view representation of yet another embodiment of a vehicle that includes a system consistent with the disclosed embodiments.

[0017] Figure 2F This is an illustrative representation of an exemplary vehicle control system consistent with the disclosed embodiments.

[0018] Figure 3A It is an illustrative representation of the interior of a vehicle, including a rearview mirror and a user interface for a vehicle imaging system, consistent with the disclosed embodiments.

[0019] Figure 3B This is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against the vehicle's windshield, consistent with the disclosed embodiments.

[0020] Figure 3C yes Figure 3B The illustrations shown are of a camera mount consistent with the disclosed embodiments, viewed from different perspectives.

[0021] Figure 3D This is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against the vehicle's windshield, consistent with the disclosed embodiments.

[0022] Figure 4 This is an exemplary block diagram of a memory configured to store instructions for performing one or more operations, consistent with the disclosed embodiments.

[0023] Figure 5A This is a flowchart illustrating an exemplary process consistent with the disclosed embodiments for evoking one or more navigation responses based on monocular image analysis.

[0024] Figure 5B This is a flowchart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a set of images, consistent with the disclosed embodiments.

[0025] Figure 5C This is a flowchart illustrating an exemplary process consistent with the disclosed embodiments for detecting road signs and / or lane geometry information in a set of images.

[0026] Figure 5D This is a flowchart illustrating an exemplary process for detecting traffic lights in a set of images, consistent with the disclosed embodiments.

[0027] Figure 5E This is a flowchart illustrating an exemplary process consistent with the disclosed embodiments for evoking one or more navigation responses based on vehicle path.

[0028] Figure 5F This is a flowchart illustrating an exemplary process consistent with the disclosed embodiments for determining whether a vehicle ahead is changing lanes.

[0029] Figure 6 This is a flowchart illustrating an exemplary process consistent with the disclosed embodiments for evoking one or more navigation responses based on stereo image analysis.

[0030] Figure 7 This is a flowchart illustrating an exemplary process consistent with the disclosed embodiments for evoking one or more navigation responses based on the analysis of three sets of images.

[0031] Figure 8 A sparse map for providing autonomous vehicle navigation is shown, consistent with the disclosed embodiments.

[0032] Figure 9A A polynomial representation of a portion of a road segment consistent with the disclosed embodiments is shown.

[0033] Figure 9B The diagram shows curves representing the target trajectories of vehicles on a specific road segment in three-dimensional space, included in a sparse map consistent with the disclosed embodiments.

[0034] Figure 10 Example landmarks that can be included in a sparse map consistent with the disclosed embodiments are shown.

[0035] Figure 11A A polynomial representation of the trajectory consistent with the disclosed embodiments is shown.

[0036] Figure 11B and Figure 11C A target trajectory along a multi-lane road, consistent with the disclosed embodiments, is shown.

[0037] Figure 11D An example road signature profile consistent with the disclosed embodiments is shown.

[0038] Figure 12 This is a schematic diagram of a system for autonomous vehicle navigation that uses crowdsourcing data received from multiple vehicles, consistent with the disclosed embodiments.

[0039] Figure 13 An example autonomous vehicle road navigation model, represented by multiple three-dimensional splines, is shown, consistent with the disclosed embodiments.

[0040] Figure 14 A map skeleton, generated based on a combination of location information from multiple drivers, is shown, consistent with the disclosed embodiments.

[0041] Figure 15 An example of two longitudinally aligned driving vehicles with example markers serving as landmarks, consistent with the disclosed embodiments, is shown.

[0042] Figure 16 An example of longitudinal alignment of multiple drives with example markers serving as landmarks, consistent with the disclosed embodiments, is shown.

[0043] Figure 17 This is a schematic diagram of a system for generating driving data using cameras, vehicles, and servers, consistent with the disclosed embodiments.

[0044] Figure 18 This is a schematic diagram of a system for crowdsourced sparse maps consistent with the disclosed embodiments.

[0045] Figure 19 This is a flowchart illustrating an exemplary process consistent with the disclosed embodiments for generating a sparse map for autonomous vehicle navigation along road segments.

[0046] Figure 20 A block diagram of a server consistent with the disclosed embodiments is shown.

[0047] Figure 21 A block diagram of a memory consistent with the disclosed embodiments is shown.

[0048] Figure 22A process for clustering vehicle trajectories associated with vehicles, consistent with the disclosed embodiments, is shown.

[0049] Figure 23 A vehicle navigation system that can be used for autonomous navigation, consistent with the disclosed embodiments, is shown.

[0050] Figure 24A , Figure 24B , Figure 24C and Figure 24D An exemplary lane marking that can be detected is shown, consistent with the disclosed embodiments.

[0051] Figure 24E An exemplary mapped lane sign is shown, consistent with the disclosed embodiments.

[0052] Figure 24F An exemplary anomaly associated with detecting lane signs is shown, consistent with the disclosed embodiments.

[0053] Figure 25A An exemplary image of the vehicle's surroundings for navigation based on mapped lane markings, consistent with the disclosed embodiments, is shown.

[0054] Figure 25B Vehicle lateral positioning correction based on mapped lane markings in a road navigation model, consistent with the disclosed embodiments, is shown.

[0055] Figure 26A This is a flowchart illustrating an exemplary process for mapping lane markings for autonomous vehicle navigation, consistent with the disclosed embodiments.

[0056] Figure 26B This is a flowchart illustrating an exemplary process for autonomously navigating a master vehicle along a road segment using mapped lane markings, consistent with the disclosed embodiments.

[0057] Figure 27 An exemplary system for providing one or more map segments to one or more vehicles, consistent with the disclosed embodiments, is shown.

[0058] Figure 28A , Figure 28B , Figure 28C and Figure 28D An exemplary potential driving envelope of a vehicle consistent with the disclosed embodiments is shown.

[0059] Figure 28E , Figure 28F , Figure 28G and Figure 28H An exemplary map tile associated with the potential driving envelope of a vehicle, consistent with the disclosed embodiments, is shown.

[0060] Figure 29A and Figure 29B Exemplary map tiles consistent with the disclosed embodiments are shown.

[0061] Figure 30 An exemplary process for obtaining map tiles, consistent with the disclosed embodiments, is shown.

[0062] Figure 31A , Figure 31B , Figure 31C and Figure 31D An exemplary process for decoding map tiles, consistent with the disclosed embodiments, is shown.

[0063] Figure 32 This is a flowchart illustrating an exemplary process consistent with the disclosed embodiments for providing one or more map segments to one or more vehicles.

[0064] Figure 33 An exemplary system for automatically generating navigation maps about one or more road segments, consistent with the disclosed embodiments, is shown.

[0065] Figure 34A , Figure 34B and Figure 34C An exemplary process for collecting navigation information, consistent with the disclosed embodiments, is shown.

[0066] Figure 35 This is a flowchart illustrating an exemplary process for automatically generating navigation maps about one or more road segments, consistent with the disclosed embodiments.

[0067] Figure 36 This is a schematic diagram of an exemplary system for providing one or more map segments to one or more vehicles, consistent with the disclosed embodiments.

[0068] Figure 37 This is a schematic diagram of an example map tile and tile segment.

[0069] Figure 38 This is a schematic diagram of an example map tile and tile segment.

[0070] Figure 39 An exemplary vehicle consistent with the disclosed embodiments is shown.

[0071] Figure 40 This is a flowchart illustrating an exemplary process for navigating a vehicle, consistent with the disclosed embodiments. Detailed Implementation

[0072] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numerals are used in the drawings and the following description to refer to the same or similar parts. While several illustrative embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, components shown in the drawings may be replaced, added, or modified, and the illustrative methods described herein may be modified by replacing, reordering, removing, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the appropriate scope is defined by the appended claims.

[0073] Overview of Autonomous Vehicles

[0074] As used throughout this disclosure, the term "autonomous vehicle" refers to a vehicle capable of making at least one navigation change without driver input. "Navigation change" refers to one or more changes in the vehicle's steering, braking, or acceleration. For autonomy to be achieved, the vehicle does not need to be fully automatic (e.g., fully operable without a driver or driver input). Rather, autonomous vehicles include those capable of operating under driver control during certain time periods and operating without driver control during other time periods. Autonomous vehicles may also include those that control only some aspects of vehicle navigation, such as steering (e.g., maintaining a vehicle path between lane restrictions), but leave other aspects to the driver (e.g., braking). In some cases, an autonomous vehicle may handle some or all of the vehicle's braking, speed control, and / or steering.

[0075] Because human drivers typically rely on visual cues and observation to control vehicles, traffic infrastructure has been constructed accordingly, with lane markings, traffic signs, and traffic lights all designed to provide visual information to the driver. Given these design characteristics of traffic infrastructure, autonomous vehicles can include cameras and processing units that analyze visual information captured from the vehicle's environment. Visual information can include, for example, components of traffic infrastructure (e.g., lane markings, traffic signs, traffic lights, etc.) that can be observed by the driver, as well as other obstacles (e.g., other vehicles, pedestrians, debris, etc.). Furthermore, autonomous vehicles can also use stored information, such as information that provides a model of the vehicle's environment during navigation. For example, a vehicle can use GPS data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data to provide information relevant to its environment while the vehicle is in motion, and the vehicle (and other vehicles) can use this information to position itself on the model.

[0076] In some embodiments of this disclosure, the autonomous vehicle may use information obtained during navigation (from cameras, GPS devices, accelerometers, speed sensors, suspension sensors, etc.). In other embodiments, the autonomous vehicle may use information obtained from past navigation of the vehicle (or other vehicles). In still other embodiments, the autonomous vehicle may use a combination of information obtained during navigation and information obtained from past navigation. The following sections provide an overview of a system consistent with the disclosed embodiments, followed by an overview of forward imaging systems and methods consistent with the system. The following sections disclose systems and methods for constructing, using, and updating sparse maps for navigation of autonomous vehicles.

[0077] System Overview

[0078] Figure 1 This is a block diagram representing system 100 consistent with the disclosed exemplary embodiments. System 100 may include various components depending on the requirements of a particular implementation. In some embodiments, system 100 may include a processing unit 110, an image acquisition unit 120, a position sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. Processing unit 110 may include one or more processing means. In some embodiments, processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing means. Similarly, image acquisition unit 120 may include any number of image acquisition means and components depending on the requirements of a particular application. In some embodiments, image acquisition unit 120 may include one or more image capture means (e.g., cameras), such as image capture means 122, image capture means 124, and image capture means 126. System 100 may also communicatively connect processing means 110 to a data interface 128 of image acquisition means 120. For example, data interface 128 may include one or more wired and / or wireless links for transmitting image data acquired by image acquisition device 120 to processing unit 110.

[0079] Wireless transceiver 172 may include one or more devices configured to exchange transmissions to one or more networks (e.g., cellular, internet, etc.) via an air interface using radio frequency, infrared frequency, magnetic field, or electric field. Wireless transceiver 172 may use any known standard to transmit and / or receive data (e.g., Wi-Fi, etc.). (Bluetooth Smart, 802.15.4, ZigBee, etc.). Such transmissions can include communication from the master vehicle to one or more remote servers. Such transmissions can also include (one-way or two-way) communication between the master vehicle and one or more target vehicles in the master vehicle's environment (e.g., to coordinate the master vehicle's navigation in view of or in conjunction with target vehicles in the master vehicle's environment), or even include broadcast transmissions to unspecified receivers near the transmitting vehicle.

[0080] Both application processor 180 and image processor 190 can include various types of hardware-based processing devices. For example, either or both of application processor 180 and image processor 190 can include a microprocessor, preprocessor (such as an image preprocessor), graphics processor, central processing unit (CPU), support circuitry, digital signal processor, integrated circuit, memory, or any other type of device suitable for running applications and for image processing and analysis. In some embodiments, application processor 180 and / or image processor 190 can include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, etc. Various processing devices can be used, including, for example, those from, such as... Processors obtained from manufacturers such as [list of manufacturers] or from [list of manufacturers]. GPUs obtained from manufacturers, and can include various architectures (e.g., x86 processors, ...). wait).

[0081] In some embodiments, application processor 180 and / or image processor 190 may include components that can be accessed from... Any EyeQ series processor chip obtained. These processor designs each contain multiple processing units with local memory and instruction sets. Such processors can include video input for receiving image data from multiple image sensors, and can also include video output capabilities. In one example, Using 90 nanometer-micron technology operating at 332 MHz. The architecture consists of two floating-point hyper-threaded 32-bit RISC CPUs ( (core), five Visual Computing Engines (VCE), and three Vector Microcode Processors The system consists of a Denali 64-bit mobile DDR controller, a 128-bit internal Sonics Interconnect, dual 16-bit video input and 18-bit video output controllers, a 16-channel DMA, and several peripherals. The MIPS34K CPU manages these five VCEs and three VMPs. TMAnd DMA, a second MIPS34K CPU and multi-channel DMA, and other peripheral devices. These five VCEs, three The MIPS34K CPU can perform the intensive vision computing required for versatile bundled applications. In another instance, it can be used as a third-generation processor in the disclosed embodiments and is superior to... Six times stronger In other examples, the disclosed embodiments may be used. and / or Of course, any updated or future EyeQ processing device can also be used with the disclosed embodiments.

[0082] Any processing device disclosed herein can be configured to perform certain functions. Configuring a processing device (such as any described EyeQ processor or other controller or microprocessor) to perform certain functions may involve programming computer-executable instructions and making those instructions available to the processing device for execution during operation of the processing device. In some embodiments, configuring the processing device may involve programming the processing device directly using architectural instructions. For example, processing devices such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., can be configured using, for example, one or more hardware description languages ​​(HDLs).

[0083] In other embodiments, the configuration processing means may include storing executable instructions on memory accessible to the processing means during operation. For example, the processing means may access the memory during operation to obtain and execute the stored instructions. In any case, the processing means configured to perform the sensing, image analysis, and / or navigation functions disclosed herein represents a hardware-based dedicated system that controls multiple hardware-based components of a host vehicle.

[0084] Although Figure 1 Two separate processing devices are depicted within processing unit 110, but more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to perform the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by two or more processing devices. Furthermore, in some embodiments, system 100 may include one or more of processing units 110 without including other components such as image acquisition unit 120.

[0085] Processing unit 110 may include various types of devices. For example, processing unit 110 may include various devices such as controllers, image preprocessors, central processing units (CPUs), graphics processing units (GPUs), support circuitry, digital signal processors, integrated circuits, memory, or any other type of device for image processing and analysis. The image preprocessor may include a video processor for capturing, digitizing, and processing images from an image sensor. The CPU may include any number of microcontrollers or microprocessors. The GPU may also include any number of microcontrollers or microprocessors. Support circuitry may be any number of circuits known in the art, including caches, power supplies, clocks, and input / output circuitry. Memory may store software that controls the operation of the system when executed by the processor. Memory may include databases and image processing software. Memory may include any number of random access memories, read-only memories, flash memory, disk drives, optical storage devices, magnetic tape storage devices, removable storage devices, and other types of storage devices. In one instance, the memory may be separate from processing unit 110. In another instance, the memory may be integrated into processing unit 110.

[0086] Each memory unit 140, 150 may contain software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), can control the operation of various aspects of system 100. These memory units may contain various databases and image processing software, as well as trained systems such as neural networks or, for example, deep neural networks. The memory units may include random access memory (RAM), read-only memory (ROM), flash memory, disk drives, optical storage devices, magnetic tape storage devices, removable storage devices, and / or any other type of storage device. In some embodiments, memory units 140, 150 may be separate from application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into application processor 180 and / or image processor 190.

[0087] The position sensor 130 may include any type of means suitable for determining the location associated with at least one component of the system 100. In some embodiments, the position sensor 130 may include a GPS receiver. Such a receiver can determine the user's location and speed by processing signals broadcast by Global Positioning System satellites. The location information from the position sensor 130 may be made available to the application processor 180 and / or the image processor 190.

[0088] In some embodiments, system 100 may include components such as a speed sensor (e.g., tachometer, speedometer) for measuring the speed of vehicle 200 and / or an accelerometer (single-axis or multi-axis) for measuring the acceleration of vehicle 200.

[0089] User interface 170 may include any means suitable for providing information to or receiving input from one or more users of system 100. In some embodiments, user interface 170 may include user input devices, including, for example, a touchscreen, microphone, keyboard, pointer device, tracking wheel, camera, knob, button, etc. Using such input devices, users can provide information input or commands to system 100 by typing instructions or information, providing voice commands, selecting menu options on the screen using buttons, pointers, or eye-tracking capabilities, or by any other technology suitable for communicating information to system 100.

[0090] User interface 170 may be equipped with one or more processing devices configured to provide and receive information from the user, and process the information for use by, for example, application processor 180. In some embodiments, such processing devices may execute instructions for recognizing and tracking eye movements, receiving and interpreting voice commands, recognizing and interpreting touches and / or gestures made on a touchscreen, responding to keyboard input or menu selections, etc. In some embodiments, user interface 170 may include a display, speakers, haptic devices, and / or any other means for providing output information to the user.

[0091] Map database 160 can contain any type of database for storing map data useful to system 100. In some embodiments, map database 160 can contain data relating to the location of various items in a reference coordinate system, including roads, water features, geographic features, commercial areas, points of interest, restaurants, gas stations, etc. Map database 160 can not only store the locations of such items but also descriptors associated with such items, including, for example, names associated with any stored features. In some embodiments, map database 160 can be physically located together with other components of system 100. Alternatively or additionally, map database 160 or a portion thereof can be located remotely relative to other components of system 100 (e.g., processing unit 110). In such embodiments, information from map database 160 can be downloaded via a wired or wireless data connection to a network (e.g., via cellular networks and / or the Internet, etc.). In some cases, map database 160 can store a sparse data model containing a polynomial representation of certain road features (e.g., lane markings) or the target trajectory of a primary vehicle. Reference is made below. Figures 8 to 19 Discuss the systems and methods for generating such maps.

[0092] Image capture devices 122, 124, and 126 may each include any type of means suitable for capturing at least one image from the environment. Furthermore, any number of image capture devices can be used to acquire images for input to an image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, or even four or more image capture devices. Reference will be made below. Figures 2B to 2E The image capture devices 122, 124 and 126 are further described.

[0093] System 100 or its various components can be integrated into a variety of different platforms. In some embodiments, system 100 may be included on vehicle 200, such as... Figure 2A As shown. For example, vehicle 200 may be equipped with the above-mentioned features. Figure 1 The system 100 includes the processing unit 110 and any other components. While in some embodiments, the vehicle 200 may be equipped with only a single image capture device (e.g., a camera), in other embodiments, such as combining... Figures 2B-2E The ones discussed can use multiple image capture devices. For example, such as Figure 2A As shown, either of the image capture devices 122 and 124 of the vehicle 200 can be part of an ADAS (Advanced Driver Assistance System) imaging set.

[0094] The image capturing device, included on the vehicle 200 and as part of the image acquisition unit 120, can be positioned at any suitable location. In some embodiments, such as Figures 2A-2E as well as Figures 3A-3C As shown, the image capture device 122 can be located near the rearview mirror. This location provides a similar line of sight to that of the driver of vehicle 200, which can help determine what is visible and invisible to the driver. The image capture device 122 can be positioned in any location near the rearview mirror, and placing the image capture device 122 on the driver's side of the mirror can further help obtain an image representing the driver's field of vision and / or line of sight.

[0095] Other positioning can also be used for the image capturing device of image acquisition unit 120. For example, image capturing device 124 can be located on or in the bumper of vehicle 200. Such positioning can be particularly suitable for image capturing devices with a wide field of view. The line of sight of the image capturing device located on the bumper may be different from that of the driver, and therefore, the bumper image capturing device and the driver may not always see the same object. Image capturing devices (e.g., image capturing devices 122, 124, and 126) can also be located in other positions. For example, the image capturing device can be located on or in one or both of the side mirrors of vehicle 200, on the roof of vehicle 200, on the hood of vehicle 200, on the trunk of vehicle 200, on the side of vehicle 200, mounted on any window of vehicle 200, positioned behind any window of vehicle 200, or positioned in front of any window of vehicle 200, and mounted in or near lights on the front and / or rear of vehicle 200, etc.

[0096] In addition to the image capture device, vehicle 200 may also include various other components of system 100. For example, processing unit 110 may be included on vehicle 200, integrated with or separate from the vehicle's engine control unit (ECU). Vehicle 200 may also be equipped with position sensor 130 such as a GPS receiver, and may also include map database 160 and memory units 140 and 150.

[0097] As previously discussed, the wireless transceiver 172 can receive data via one or more networks (e.g., cellular networks, the Internet, etc.) and / or. For example, the wireless transceiver 172 can upload data collected by the system 100 to one or more servers and download data from one or more servers. For example, via the wireless transceiver 172, the system 100 can receive periodic or on-demand updates to data stored in map database 160, memory 140, and / or memory 150. Similarly, the wireless transceiver 172 can upload any data from the system 100 (e.g., images captured by image acquisition unit 120, data received by position sensor 130 or other sensors, vehicle control system, etc.) and / or any data processed by processing unit 110 to one or more servers.

[0098] System 100 can upload data to a server (e.g., to the cloud) based on privacy level settings. For example, System 100 can implement privacy level settings to specify or restrict the types of data (including metadata) that can uniquely identify the vehicle and / or the vehicle's driver / owner that are transmitted to the server. Such settings can be configured by the user via, for example, a wireless transceiver 172, and can be initialized by factory default settings or data received by the wireless transceiver 172.

[0099] In some embodiments, system 100 may upload data according to a “high” privacy level, and under certain settings, system 100 may transmit data (e.g., route-related location information, captured images, etc.) without any details about a specific vehicle and / or driver / owner. For example, when uploading data according to a “high” privacy setting, system 100 may omit the vehicle identification number (VIN) or the name of the vehicle’s driver or owner, and may instead transmit data such as captured images and / or restricted route-related location information.

[0100] Other privacy levels are anticipated. For example, system 100 may transmit data to the server at an “intermediate” privacy level and may include additional information not included at a “high” privacy level, such as the vehicle’s brand and / or model and / or vehicle type (e.g., passenger car, SUV, truck, etc.). In some embodiments, system 100 may upload data at a “low” privacy level. At a “low” privacy level, system 100 may upload data containing information sufficient to uniquely identify a specific vehicle, owner / driver, and / or part or all of the route traveled by the vehicle. Such “low” privacy level data may include one or more of the following: for example, VIN, driver / owner name, vehicle’s origin point before departure, vehicle’s intended destination, vehicle’s brand and / or model, vehicle type, etc.

[0101] Figure 2A This is an illustrative side view representation of an exemplary vehicle imaging system consistent with the disclosed embodiments. Figure 2B yes Figure 2A The illustrated top view of the embodiment is shown. Figure 2B As shown, the disclosed embodiments may include a vehicle 200, which includes a system 100 in its body, the system 100 having a first image capturing device 122 positioned near and / or close to the driver's side mirror of the vehicle 200, a second image capturing device 124 positioned on or in the bumper area of ​​the vehicle 200 (e.g., one of the bumper areas 210), and a processing unit 110.

[0102] like Figure 2C As shown, both image capture devices 122 and 124 can be positioned near the rearview mirror of vehicle 200 and / or close to the driver. Furthermore, although... Figure 2B and Figure 2C Two image capture devices 122 and 124 are shown; it should be understood that other embodiments may include more than two image capture devices. For example, in Figure 2D and 2E In the embodiment shown, the first image capture device 122, the second image capture device 124, and the third image capture device 126 are included in the system 100 of the vehicle 200.

[0103] like Figure 2D As shown, image capture device 122 can be positioned near and / or close to the rearview mirror of vehicle 200, and image capture devices 124 and 126 can be positioned above or in the bumper area of ​​vehicle 200 (e.g., one of bumper areas 210). And as... Figure 2E As shown, image capturing devices 122, 124, and 126 can be positioned near the rearview mirror and / or close to the driver's seat of vehicle 200. The disclosed embodiments are not limited to any particular number and configuration of image capturing devices, and the image capturing devices can be positioned in any suitable location within or on vehicle 200.

[0104] It should be understood that the disclosed embodiments are not limited to vehicles and can be applied to other scenarios. It should also be understood that the disclosed embodiments are not limited to a specific type of vehicle 200, but can be applied to all types of vehicles, including cars, trucks, trailers, and other types of vehicles.

[0105] The first image capturing device 122 can comprise any suitable type of image capturing device. The image capturing device 122 can include an optical axis. In one example, the image capturing device 122 can include an Aptina M9V024WVGA sensor with a global shutter. In other embodiments, the image capturing device 122 can provide a resolution of 1280 × 960 pixels and can include a rolling shutter. The image capturing device 122 can include various optical elements. In some embodiments, it can include one or more lenses, for example, for providing the desired focal length and field of view for the image capturing device. In some embodiments, the image capturing device 122 can be associated with a 6mm lens or a 12mm lens. In some embodiments, such as Figure 2DAs shown, the image capture device 122 can be configured to capture images with a desired field of view (FOV) 202. For example, the image capture device 122 can be configured to have a conventional FOV, such as a 46-degree FOV, 50-degree FOV, 52-degree FOV, or larger, within the range of 40 to 56 degrees. Alternatively, the image capture device 122 can be configured to have a narrow FOV, such as a 28-degree FOV or a 36-degree FOV, within the range of 23 to 40 degrees. Additionally, the image capture device 122 can be configured to have a wide FOV within the range of 100 to 180 degrees. In some embodiments, the image capture device 122 can include a wide-angle bumper camera or a camera with an FOV of up to 180 degrees. In some embodiments, the image capture device 122 can be a 7.2-megapixel image capture device with an aspect ratio of approximately 2:1 (e.g., HxV = 3800 × 1900 pixels) and a horizontal FOV of approximately 100 degrees. Such an image capture device can be used instead of a three-image capture device configuration. Due to significant lens distortion, in embodiments using radially symmetrical lenses, the vertical FOV of such an image capture device can be significantly less than 50 degrees. For example, such a lens may not be radially symmetrical, which would allow a vertical FOV greater than 50 degrees with a horizontal FOV of 100 degrees.

[0106] The first image capturing device 122 can acquire multiple first images of a scene associated with the vehicle 200. Each of the multiple first images can be acquired as a series of image scan lines, which can be captured using a rolling shutter. Each scan line can contain multiple pixels.

[0107] The first image capture device 122 may have a scan rate associated with the acquisition of each of the first series of image scan lines. The scan rate may refer to the rate at which the image sensor can acquire image data associated with each pixel contained in a particular scan line.

[0108] Image capture devices 122, 124, and 126 may contain any suitable type and number of image sensors, such as CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be used in conjunction with a rolling shutter, such that each pixel in a line is read one at a time, and the scanning of the lines is performed on a line-by-line basis until the entire image frame has been captured. In some embodiments, lines may be captured sequentially from top to bottom relative to the frame.

[0109] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124, and 126) may constitute a high-resolution imager and may have a resolution greater than 5M pixels, 7M pixels, 10M pixels, or more pixels.

[0110] The use of a rolling shutter can cause pixels in different rows to be exposed and captured at different times, which can cause skew and other image artifacts in the captured image frame. On the other hand, when the image capture device 122 is configured to utilize global or synchronous shutter operation, all pixels can be exposed for the same amount of time and during a common exposure period. As a result, the image data in the frame collected from a system using a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a specific time. In contrast, in a rolling shutter application, each row in the frame is exposed and data is captured at different times. Therefore, in an image capture device with a rolling shutter, moving objects may be distorted. This phenomenon will be described in more detail below.

[0111] The second image capture device 124 and the third image capture device 126 can be any type of image capture device. Similar to the first image capture device 122, each of image capture devices 124 and 126 can include an optical axis. In one embodiment, each of image capture devices 124 and 126 can include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of image capture devices 124 and 126 can include a rolling shutter. Similar to image capture device 122, image capture devices 124 and 126 can be configured to include various lenses and optics. In some embodiments, the lenses associated with image capture devices 124 and 126 can provide a field of view (FOV) (such as FOV 204 and 206) that is equal to or narrower than the FOV associated with image capture device 122 (such as FOV 202). For example, image capture devices 124 and 126 can have an FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less.

[0112] Image capture devices 124 and 126 can acquire multiple second and third images of a scene associated with vehicle 200. Each of the multiple second and third images can be acquired as a second and third series of image scan lines, which can be captured using a rolling shutter. Each scan line or row can have multiple pixels. Image capture devices 124 and 126 can have second and third scan rates associated with the acquisition of each image scan line contained in the second and third series.

[0113] Each image capture device 122, 124, and 126 can be positioned at any suitable location and orientation relative to the vehicle 200. The relative positions of the image capture devices 122, 124, and 126 can be selected to facilitate the fusion of information acquired from the image capture devices. For example, in some embodiments, the field of view (FOV) associated with image capture device 124 (such as FOV 204) may partially or completely overlap with the FOV associated with image capture device 122 (e.g., FOV 202) and the FOV associated with image capture device 126 (e.g., FOV 206).

[0114] Image capture devices 122, 124, and 126 can be located at any suitable relative height on vehicle 200. In one example, a height difference may exist between image capture devices 122, 124, and 126, which can provide sufficient parallax information to enable stereoscopic analysis. For example, as Figure 2A As shown, the two image capture devices 122 and 124 are at different heights. For example, a lateral displacement difference may also exist between image capture devices 122, 124, and 126 to provide additional parallax information for the stereo analysis of the processing unit 110. The difference in lateral displacement can be represented by dx, such as... Figure 2C and Figure 2D As shown. In some embodiments, there may be forward or backward displacement (e.g., range displacement) between image capture devices 122, 124, and 126. For example, image capture device 122 may be located 0.5 to 2 meters or more behind image capture devices 124 and / or 126. This type of displacement allows one of the image capture devices to cover potential blind spots of the other(s) image capture devices(s).

[0115] Image capture device 122 may have any suitable resolution capability (e.g., the number of pixels associated with an image sensor), and the resolution of one or more image sensors associated with image capture device 122 may be higher, lower, or the same as the resolution of one or more image sensors associated with image capture devices 124 and 126. In some embodiments, one or more image sensors associated with image capture device 122 and / or image capture devices 124 and 126 may have a resolution of 640×480, 1024×768, 1280×960, or any other suitable resolution.

[0116] The frame rate (e.g., the rate at which an image capture device acquires a set of pixel data for an image frame before continuing to capture pixel data associated with the next image frame) can be controllable. The frame rate associated with image capture device 122 can be higher, lower, or the same as the frame rates associated with image capture devices 124 and 126. The frame rates associated with image capture devices 122, 124, and 126 can depend on various factors that may affect the timing of the frame rate. For example, one or more of image capture devices 122, 124, and 126 may include selectable pixel delay periods that are applied before or after acquiring image data associated with one or more pixels of the image sensors in image capture devices 122, 124, and / or 126. Typically, image data corresponding to each pixel can be acquired according to the clock rate used for the device (e.g., one pixel per clock cycle). Furthermore, in embodiments including a rolling shutter, one or more of the image capture devices 122, 124, and 126 may include a selectable horizontal blanking period applied before or after acquiring image data associated with a row of pixels of the image sensors in the image capture devices 122, 124, and / or 126. Additionally, one or more of the image capture devices 122, 124, and / or 126 may include a selectable vertical blanking period applied before or after acquiring image data associated with image frames of the image capture devices 122, 124, and 126.

[0117] These timing controls enable synchronization of the frame rates associated with image capture devices 122, 124, and 126, even if each has a different line scan rate. Furthermore, as will be discussed in more detail below, these selectable timing controls, along with other factors (e.g., image sensor resolution, maximum line scan rate, etc.), enable synchronization of image capture from areas where the field of view (FOV) of image capture device 122 overlaps with one or more FOVs of image capture devices 124 and 126, even if the field of view of image capture device 122 differs from the FOVs of image capture devices 124 and 126.

[0118] The frame rate timing in the image capture devices 122, 124, and 126 can depend on the resolution of the associated image sensor. For example, assuming that the line scan rates are similar for two devices, if one device contains an image sensor with a resolution of 640×480 and the other device contains an image sensor with a resolution of 1280×960, then more time is required to acquire one frame of image data from the sensor with the higher resolution.

[0119] Another factor that can affect the timing of image data acquisition in image capture devices 122, 124, and 126 is the maximum line scan rate. For example, acquiring one line of image data from an image sensor included in image capture devices 122, 124, and 126 will require a certain minimum amount of time. Assuming no pixel delay period is added, this minimum amount of time for acquiring one line of image data will be related to the maximum line scan rate for a particular device. Devices providing a higher maximum line scan rate have the potential to provide a higher frame rate than devices with a lower maximum line scan rate. In some embodiments, one or more of image capture devices 124 and 126 may have a maximum line scan rate higher than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture devices 124 and / or 126 may be 1.25, 1.5, 1.75, or 2 times or more of the maximum line scan rate of image capture device 122.

[0120] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may operate at a scan rate less than or equal to its maximum scan rate. The system may be configured such that one or more of image capture devices 124 and 126 operate at a line scan rate equal to the line scan rate of image capture device 122. In other instances, the system may be configured such that the line scan rate of image capture device 124 and / or image capture device 126 may be 1.25, 1.5, 1.75, or 2 or more times the line scan rate of image capture device 122.

[0121] In some embodiments, image capturing devices 122, 124, and 126 may be asymmetrical. In other words, they may comprise cameras with different fields of view (FOV) and focal lengths. For example, the fields of view of image capturing devices 122, 124, and 126 may encompass any desired area of ​​the environment surrounding vehicle 200. In some embodiments, one or more of image capturing devices 122, 124, and 126 may be configured to acquire image data from the environment in front of, behind, to the side of, or a combination thereof surrounding vehicle 200.

[0122] Furthermore, the focal length associated with each image capturing device 122, 124, and / or 126 can be selectable (e.g., by including a suitable lens, etc.) so that each device acquires an image of an object at a desired distance range relative to the vehicle 200. For example, in some embodiments, image capturing devices 122, 124, and 126 can acquire images of approaching objects within a few meters of the vehicle. Image capturing devices 122, 124, and 126 can also be configured to acquire images of objects at a greater distance from the vehicle (e.g., 25 meters, 50 meters, 100 meters, 150 meters, or more). Furthermore, the focal lengths of image capturing devices 122, 124, and 126 can be selected such that one image capturing device (e.g., image capturing device 122) can acquire images of objects relatively close to the vehicle (e.g., within 10 or 20 meters), while other image capturing devices (e.g., image capturing devices 124 and 126) can acquire images of objects farther away from the vehicle 200 (e.g., greater than 20 meters, 50 meters, 100 meters, 150 meters, etc.).

[0123] According to some embodiments, the field of view (FOV) of one or more image capture devices 122, 124, and 126 may have a wide angle. For example, an FOV of 140 degrees may be advantageous, especially for image capture devices 122, 124, and 126 that can be used to capture images of areas near vehicle 200. For example, image capture device 122 may be used to capture images of areas to the right or left of vehicle 200, and in such embodiments, it may be desirable for image capture device 122 to have a wide FOV (e.g., at least 140 degrees).

[0124] The field of view associated with each of the image capturing devices 122, 124, and 126 can depend on the corresponding focal length. For example, as the focal length increases, the corresponding field of view decreases.

[0125] Image capture devices 122, 124, and 126 can be configured to have any suitable field of view. In one particular example, image capture device 122 may have a horizontal FOV of 46 degrees, image capture device 124 may have a horizontal FOV of 23 degrees, and image capture device 126 may have a horizontal FOV between 23 degrees and 46 degrees. In another particular example, image capture device 122 may have a horizontal FOV of 52 degrees, image capture device 124 may have a horizontal FOV of 26 degrees, and image capture device 126 may have a horizontal FOV between 26 degrees and 52 degrees. In some embodiments, the ratio of the FOV of image capture device 122 to the FOV of image capture device 124 and / or image capture device 126 may vary from 1.5 to 2.0. In other embodiments, this ratio may vary between 1.25 and 2.25.

[0126] System 100 may be configured such that the field of view of image capture device 122 at least partially or completely overlaps with the field of view of image capture devices 124 and / or 126. In some embodiments, system 100 may be configured such that the field of view of image capture devices 124 and 126, for example, falls within (e.g., is narrower than) the field of view of image capture device 122 and shares a common center with the field of view of image capture device 122. In other embodiments, image capture devices 122, 124, and 126 may capture adjacent FOVs, or may have partial overlap in their FOVs. In some embodiments, the field of view of image capture devices 122, 124, and 126 may be aligned such that the center of the narrower FOV image capture device 124 and / or 126 may be located in the lower half of the field of view of the wider FOV device 122.

[0127] Figure 2F This is a schematic representation of an exemplary vehicle control system consistent with the disclosed embodiments. (As...) Figure 2F As indicated, vehicle 200 may include a throttle control system 220, a braking system 230, and a steering system 240. System 100 may provide inputs (e.g., control signals) to one or more of the throttle control system 220, braking system 230, and steering system 240 via one or more data links (e.g., any wired and / or wireless links for transmitting data). For example, based on analysis of images acquired by image capture devices 122, 124, and / or 126, system 100 may provide control signals to one or more of the throttle control system 220, braking system 230, and steering system 240 to navigate vehicle 200 (e.g., by inducing acceleration, steering, lane shift, etc.). Furthermore, system 100 may receive inputs from one or more of the throttle control system 220, braking system 230, and steering system 240 indicating the operating conditions of vehicle 200 (e.g., speed, whether vehicle 200 is braking and / or steering, etc.). The following is combined with... Figures 4 to 7 Further details will be provided.

[0128] like Figure 3AAs shown, vehicle 200 may also include a user interface 170 for interacting with the driver or passengers of vehicle 200. For example, user interface 170 in a vehicle application may include a touchscreen 320, a knob 330, a button 340, and a microphone 350. The driver or passengers of vehicle 200 may also interact with system 100 using handles (e.g., located on or near the steering wheel of vehicle 200, including, for example, a turn signal handle), buttons (e.g., located on the steering wheel of vehicle 200), etc. In some embodiments, microphone 350 may be positioned adjacent to rearview mirror 310. Similarly, in some embodiments, image capture device 122 may be located near rearview mirror 310. In some embodiments, user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, system 100 may provide various notifications (e.g., alarms) via speaker 360.

[0129] Figures 3B to 3D This is an illustration of an exemplary camera mount 370 configured to be positioned behind a rearview mirror (e.g., rearview mirror 310) and against the vehicle windshield, consistent with the disclosed embodiments. Figure 3B As shown, camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be positioned behind a light shield 380, which may be flush with the vehicle windshield and comprise a composite of a film and / or anti-reflective material. For example, the light shield 380 may be positioned such that the shield is aligned with a vehicle windshield having a matching bevel. In some embodiments, each of the image capture devices 122, 124, and 126 may be positioned behind the light shield 380, for example, in Figure 3D The embodiments described herein are not limited to any particular configuration of the image capture devices 122, 124 and 126, the camera mount 370 and the light shield 380. Figure 3C yes Figure 3B The image shown is a front view of the camera mount 370.

[0130] As those skilled in the art who benefit from this disclosure will understand, many variations and / or modifications can be made to the disclosed embodiments. For example, not all components are necessary for the operation of system 100. Furthermore, in providing the functionality of the disclosed embodiments, any component may be located in any suitable part of system 100 and components may be rearranged into various configurations. Thus, the foregoing configurations are exemplary, and regardless of the configurations discussed above, system 100 can provide a wide range of functions to analyze the surroundings of vehicle 200 and navigate vehicle 200 in response to that analysis.

[0131] As discussed in more detail below and according to various disclosed embodiments, system 100 can provide various features related to autonomous driving and / or driver assistance technologies. For example, system 100 can analyze image data, location data (e.g., GPS positioning information), map data, speed data, and / or data from sensors included in vehicle 200. System 100 can collect data from, for example, image acquisition unit 120, position sensor 130, and other sensors for analysis. Furthermore, system 100 can analyze the collected data to determine whether vehicle 200 should take a certain action, and then automatically take the determined action without human intervention. For example, when vehicle 200 is navigating without human intervention, system 100 can automatically control the braking, acceleration, and / or steering of vehicle 200 (e.g., by transmitting control signals to one or more of throttle adjustment system 220, braking system 230, and steering system 240). Additionally, system 100 can analyze the collected data and issue warnings and / or alerts to vehicle occupants based on the analysis of the collected data. Additional details regarding various embodiments of system 100 are provided below.

[0132] Forward Multi-Imaging System

[0133] As discussed above, system 100 can provide driver assistance functions using a multi-camera system. The multi-camera system may use one or more cameras facing forward of the vehicle. In other embodiments, the multi-camera system may include one or more cameras facing the side or rear of the vehicle. In one embodiment, for example, system 100 may use a dual-camera imaging system, wherein a first camera and a second camera (e.g., image capture devices 122 and 124) may be positioned at the front and / or side of the vehicle (e.g., vehicle 200). The first camera may have a field of view greater than, less than, or partially overlapping with the field of view of the second camera. Furthermore, the first camera may be connected to a first image processor for monocular image analysis of images provided by the first camera, and the second camera may be connected to a second image processor for monocular image analysis of images provided by the second camera. The outputs of the first and second image processors (e.g., processed information) may be combined. In some embodiments, the second image processor may receive images from both the first and second cameras to perform stereo analysis. In another embodiment, system 100 may use a three-camera imaging system, wherein each camera has a different field of view. Therefore, such a system can make decisions based on information derived from objects located at different distances in front of and to the sides of the vehicle. Reference for monocular image analysis can be to instances of image analysis based on images captured from a single viewpoint (e.g., from a single camera). Stereo image analysis can be to instances of image analysis based on two or more images captured using one or more variations of image capture parameters. For example, images suitable for stereo image analysis could include images captured from two or more different locations, from different fields of view, using different focal lengths, and along with parallax information.

[0134] For example, in one embodiment, system 100 may use image capture devices 122, 124, and 126 to implement a three-camera configuration. In such a configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees or other values ​​selected from the range of approximately 20 to 45 degrees), image capture device 124 may provide a wide field of view (e.g., 150 degrees or other values ​​selected from the range of approximately 100 to approximately 180 degrees), and image capture device 126 may provide an intermediate field of view (e.g., 46 degrees or other values ​​selected from the range of approximately 35 to approximately 60 degrees). In some embodiments, image capture device 126 may serve as a main camera or a base camera. Image capture devices 122, 124, and 126 may be positioned behind rearview mirror 310 and substantially side-by-side (e.g., 6 cm apart). Furthermore, in some embodiments, as discussed above, one or more of image capture devices 122, 124, and 126 may be mounted behind a sun visor 380 flush with the windshield of vehicle 200. Such occlusion can be used to minimize the impact of any reflections from inside the car on the image capture devices 122, 124 and 126.

[0135] In another embodiment, as described above Figure 3B and Figure 3C The wide field-of-view camera discussed above (e.g., image capture device 124 in the example above) can be mounted below the narrow field-of-view camera and the main field-of-view camera (e.g., image capture devices 122 and 126 in the example above). Such a configuration can provide a free line of sight from the wide field-of-view camera. To reduce reflections, the camera can be mounted close to the windshield of the vehicle 200, and a polarizer can be included on the camera to dampen reflected light.

[0136] A three-camera system can provide certain performance characteristics. For example, some embodiments may include the ability of one camera to verify the detection of an object based on the detection results from another camera. In the three-camera configuration discussed above, the processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips as discussed above), wherein each processing device is dedicated to processing images captured by one or more of the image capture devices 122, 124, and 126.

[0137] In a three-camera system, the first processing unit can receive images from both the main camera and the narrow field-of-view camera, and perform visual processing on the narrow FOV camera, such as detecting other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Furthermore, the first processing unit can calculate the pixel parallax between the images from the main camera and the narrow camera, and create a 3D reconstruction of the vehicle 200's environment. The first processing unit can then combine the 3D reconstruction with 3D map data, or combine the 3D reconstruction with 3D information calculated based on information from the other camera.

[0138] The second processing unit can receive images from the main camera and perform visual processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. Furthermore, the second processing unit can calculate camera displacement and, based on this displacement, calculate the parallax of pixels between consecutive images and create a 3D reconstruction of the scene (e.g., a structure from motion). The second processing unit can then transmit the 3D-reconstructed structure from motion to the first processing unit for combination with the stereoscopic 3D images.

[0139] The third processing unit can receive images from a wide field of view (FOV) camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing unit can further execute additional processing instructions to analyze the images to identify moving objects in the images, such as vehicles changing lanes or pedestrians.

[0140] In some embodiments, allowing the image-based information stream to be captured and processed independently can provide opportunities for redundancy in the system. Such redundancy may include, for example, using a first image capture device and images processed from that device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.

[0141] In some embodiments, system 100 may use two image capture devices (e.g., image capture devices 122 and 124) to provide navigation assistance to vehicle 200, and a third image capture device (e.g., image capture device 126) to provide redundancy and verify the analysis of data received from the other two image capture devices. For example, in such a configuration, image capture devices 122 and 124 may provide images for stereo analysis by system 100 for navigating vehicle 200, while image capture device 126 may provide images for monocular analysis by system 100 to provide redundancy and verification based on information obtained from images captured by image capture devices 122 and / or 124. In other words, image capture device 126 (and the corresponding processing device) can be considered to provide a redundant subsystem for checking the analysis derived from image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AEB) system). Furthermore, in some embodiments, redundancy and verification of received data may be supplemented based on information received from one or more sensors (e.g., radar, lidar, acoustic sensors, information received from one or more transceivers outside the vehicle, etc.).

[0142] Those skilled in the art will recognize that the camera configurations, camera placements, number of cameras, camera positioning, etc., described above are merely examples. These components, and other components described regarding the overall system, can be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of a multi-camera system to provide driver assistance and / or autonomous vehicle functionality are as follows.

[0143] Figure 4 This is an exemplary functional block diagram of memory 140 and / or memory 150, which may store / be programmed instructions for performing one or more operations consistent with embodiments of this disclosure. Although memory 140 is referred to below, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0144] like Figure 4 As shown, memory 140 may store monocular image analysis module 402, stereo image analysis module 404, velocity and acceleration module 406, and navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Furthermore, application processor 180 and / or image processor 190 may execute instructions stored in any of the modules 402, 404, 406, and 408 contained in memory 140. Those skilled in the art will understand that references to processing unit 110 in the following discussion may refer separately or jointly to application processor 180 and image processor 190. Accordingly, any of the steps of the following processes may be performed by one or more processing means.

[0145] In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform monocular image analysis on a set of images acquired by one of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensory information (e.g., information from radar, lidar, etc.) to perform monocular image analysis. As combined below... Figures 5A to 5D As described, the monocular image analysis module 402 may include instructions for detecting a set of features within a set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other features associated with the vehicle's environment. Based on this analysis, system 100 (e.g., via processing unit 110) may induce one or more navigation responses in vehicle 200, such as steering, lane changing, changes in acceleration, etc., as discussed below in conjunction with navigation response module 408.

[0146] In one embodiment, the stereo image analysis module 404 may store instructions (such as computer vision software) that, when executed by the processing unit 110, perform stereo image analysis on a first set and a second set of images acquired by a combination of image capture devices selected from any of the image capture devices 122, 124, and 126. In some embodiments, the processing unit 110 may combine information from the first set and the second set of images with additional sensory information (e.g., information from radar) to perform stereo image analysis. For example, the stereo image analysis module 404 may include instructions for performing stereo image analysis based on the first set of images acquired by the image capture device 124 and the second set of images acquired by the image capture device 126. (See below for further details.) Figure 6 As described, the stereo image analysis module 404 may include instructions for detecting a set of features within a first and second set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, etc. Based on this analysis, the processing unit 110 may induce one or more navigation responses in the vehicle 200, such as steering, lane changing, changes in acceleration, etc., as discussed below in conjunction with the navigation response module 408. Furthermore, in some embodiments, the stereo image analysis module 404 may implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems, such as systems configured to use computer vision algorithms to detect and / or label objects in the environment from which sensory information is captured and processed. In one embodiment, the stereo image analysis module 404 and / or other image processing modules may be configured to use a combination of trained and untrained systems.

[0147] In one embodiment, the speed and acceleration module 406 may store software configured to analyze data received from one or more computing and electromechanical devices in the vehicle 200, which are configured to cause changes in the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 may execute instructions associated with the speed and acceleration module 406 to calculate a target speed of the vehicle 200 based on data derived from the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include, for example, target position, speed and / or acceleration, the position and / or speed of the vehicle 200 relative to nearby vehicles, pedestrians, or road objects, and position information of the vehicle 200 relative to lane markings on the road. Additionally, the processing unit 110 may calculate the target speed of the vehicle 200 based on sensory input (e.g., information from radar) and input from other systems of the vehicle 200 (such as the throttle control system 220, braking system 230, and / or steering system 240). Based on the calculated target speed, the processing unit 110 can transmit electrical signals to the throttle adjustment system 220, braking system 230 and / or steering system 240 of the vehicle 200 to trigger a change in speed and / or acceleration by, for example, physically pressing down the brake or releasing the accelerator of the vehicle 200.

[0148] In one embodiment, the navigation response module 408 may store software that can be executed by the processing unit 110 to determine the desired navigation response based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include position and speed information associated with nearby vehicles, pedestrians, and road objects, target position information of vehicle 200, etc. Furthermore, in some embodiments, the navigation response may be (partially or entirely) based on map data, the predetermined position of vehicle 200, and / or the relative velocity or relative acceleration between vehicle 200 and one or more objects detected from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 may also be configured to determine the desired navigation response based on sensory input (e.g., information from radar) and input from other systems of vehicle 200 (such as the throttle control system 220, braking system 230, and steering system 240 of vehicle 200). Based on the desired navigation response, the processing unit 110 can transmit electrical signals to the throttle adjustment system 220, braking system 230, and steering system 240 of the vehicle 200 to achieve a predetermined angle of rotation, for example, by turning the steering wheel of the vehicle 200, thereby triggering the desired navigation response. In some embodiments, the processing unit 110 can use the output of the navigation response module 408 (e.g., the desired navigation response) as an input to the speed and acceleration module 406 to calculate the change in speed of the vehicle 200.

[0149] Furthermore, any of the modules disclosed herein (e.g., modules 402, 404, and 406) can implement techniques associated with trained systems (such as neural networks or deep neural networks) or untrained systems.

[0150] Figure 5A This is a flowchart illustrating an exemplary process 500A for inducing one or more navigation responses based on monocular image analysis, consistent with the disclosed embodiments. At step 510, processing unit 110 may receive multiple images via data interface 128 between processing unit 110 and image acquisition unit 120. For example, a camera included in image acquisition unit 120 (such as image capture device 122 having a field of view 202) may capture multiple images of a region in front of vehicle 200 (e.g., or to the side or rear of the vehicle) and transmit them to processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). At step 520, processing unit 110 may execute monocular image analysis module 402 to analyze the multiple images, as described below. Figures 5B to 5D As described in further detail. By performing this analysis, the processing unit 110 can detect a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, etc.

[0151] At step 520, processing unit 110 may also execute monocular image analysis module 402 to detect various road hazards, such as components of truck tires, fallen road signs, loose cargo, small animals, etc. Road hazards may vary in structure, shape, size, and color, which may make the detection of such hazards more difficult. In some embodiments, processing unit 110 may execute monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, processing unit 110 may estimate camera motion between consecutive image frames and calculate disparity in pixels between frames to construct a 3D map of the road. Processing unit 110 can then use this 3D map to detect the road surface and hazards present on the road surface.

[0152] At step 530, processing unit 110 may execute navigation response module 408 to perform the analysis performed at step 520 and as described above. Figure 4The described techniques are used to induce one or more navigation responses. Navigation responses may include, for example, steering, lane changing, braking, changes in acceleration, etc. In some embodiments, processing unit 110 may induce the one or more navigation responses using data derived from execution speed and acceleration module 406. Furthermore, multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof. For example, processing unit 110 may cause vehicle 200 to change lanes and then accelerate by, for example, sequentially transmitting control signals to steering system 240 and throttle adjustment system 220 of vehicle 200. Alternatively, processing unit 110 may cause vehicle 200 to brake while changing lanes by, for example, simultaneously transmitting control signals to braking system 230 and steering system 240 of vehicle 200.

[0153] Figure 5B This is a flowchart illustrating an exemplary process 500B for detecting one or more vehicles and / or pedestrians in a set of images, consistent with the disclosed embodiments. Processing unit 110 may implement process 500B by executing monocular image analysis module 402. At step 540, processing unit 110 may determine a set of candidate objects representing possible vehicles and / or pedestrians. For example, processing unit 110 may scan one or more images, compare the images with one or more predetermined patterns, and identify possible locations within each image that may contain objects of interest (e.g., vehicles, pedestrians, or portions thereof). The predetermined patterns may be designed in a way that achieves a high "false hit" rate and a low "missed" rate. For example, processing unit 110 may use a low similarity threshold on the predetermined patterns to identify candidate objects as possible vehicles or pedestrians. Doing so allows processing unit 110 to reduce the likelihood of missing (e.g., not identifying) candidate objects representing vehicles or pedestrians.

[0154] At step 542, processing unit 110 may filter the group of candidate objects based on classification criteria to exclude certain candidates (e.g., irrelevant or less relevant objects). Such criteria may be derived from various attributes associated with object type categories stored in a database (e.g., a database stored in memory 140). Attributes may include object shape, size, texture, location (e.g., relative to vehicle 200), etc. Therefore, processing unit 110 may use one or more sets of criteria to reject false candidates from the group of candidate objects.

[0155] At step 544, processing unit 110 may analyze multiple frames of images to determine whether an object in the group of candidate objects represents a vehicle and / or a pedestrian. For example, processing unit 110 may track detected candidate objects across consecutive frames and accumulate frame-by-frame data associated with the detected objects (e.g., size, position relative to vehicle 200, etc.). Furthermore, processing unit 110 may estimate parameters of the detected objects and compare the object's frame-by-frame position data with its predicted position.

[0156] At step 546, processing unit 110 can construct a set of measurements for the detected objects. Such measurements may include, for example, position, velocity, and acceleration values ​​(relative to vehicle 200) associated with the detected objects. In some embodiments, processing unit 110 may construct these measurements based on estimation techniques such as Kalman filters or linear quadratic estimation (LQE) using a series of time-based observations, or based on modeling data available for different object categories (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). The Kalman filter may be based on a measurement of the object's scale, where the scale measurement is proportional to the time of collision (e.g., the amount of time it takes for vehicle 200 to arrive at the object). Thus, by performing steps 540 through 546, processing unit 110 can identify vehicles and pedestrians appearing within the set of captured images and derive information associated with those vehicles and pedestrians (e.g., position, velocity, size). Based on this identification and the derived information, processing unit 110 may induce one or more navigation responses in vehicle 200, as described above. Figure 5A As described.

[0157] At step 548, processing unit 110 may perform optical flow analysis on one or more images to reduce the likelihood of detecting "false hits" and missing candidate objects representing vehicles or pedestrians. Optical flow analysis can refer to, for example, analyzing motion patterns relative to vehicle 200, associated with other vehicles and pedestrians, and distinct from road surface motion in one or more images. Processing unit 110 can calculate the motion of candidate objects by observing different positions of the objects across multiple image frames captured at different times. Processing unit 110 can use this position and time value as input to a mathematical model for calculating the motion of candidate objects. Therefore, optical flow analysis can provide another method for detecting vehicles and pedestrians near vehicle 200. Processing unit 110 may combine steps 540 to 546 to perform optical flow analysis to provide redundancy for detecting vehicles and pedestrians and improve the reliability of system 100.

[0158] Figure 5CThis is a flowchart illustrating an exemplary process 500C for detecting road signs and / or lane geometry information in a set of images, consistent with the disclosed embodiments. Processing unit 110 may implement process 500C by executing monocular image analysis module 402. At step 550, processing unit 110 may detect a set of objects by scanning one or more images. To detect road segments containing lane signs, lane geometry information, and other relevant road signs, processing unit 110 may filter the group of objects to exclude objects determined to be irrelevant (e.g., potholes, pebbles, etc.). At step 552, processing unit 110 may group road segments detected in step 550 that belong to the same road sign or lane sign together. Based on this grouping, processing unit 110 may generate a model, such as a mathematical model, representing the detected road segments.

[0159] At step 554, processing unit 110 may construct a set of measurements associated with the detected road segment. In some embodiments, processing unit 110 may create a projection of the detected road segment from the image plane to the real-world plane. This projection may be characterized using a cubic polynomial with coefficients corresponding to physical properties such as the location, slope, curvature, and derivative of curvature of the detected road. When generating the projection, processing unit 110 may consider changes in the road surface, as well as the pitch and roll rates associated with vehicle 200. Furthermore, processing unit 110 may model the road elevation by analyzing positional and motion cues present on the road surface. Additionally, processing unit 110 may estimate the pitch and roll rates associated with vehicle 200 by tracking a set of feature points in one or more images.

[0160] At step 556, processing unit 110 can perform multi-frame analysis, for example, by tracking detected road segments across consecutive image frames and accumulating frame-by-frame data associated with the detected road segments. Because processing unit 110 performs multi-frame analysis, the set of measurements constructed in step 554 can become more reliable and correlated with increasingly higher confidence levels. Therefore, by performing steps 550, 552, 554, and 556, processing unit 110 can identify road signs appearing in the set of captured images and derive lane geometry information. Based on this identification and the derived information, processing unit 110 can induce one or more navigation responses in vehicle 200, as described above. Figure 5A As described.

[0161] At step 558, processing unit 110 may consider additional information sources to further generate a safety model of vehicle 200 in its surrounding environment. Processing unit 110 may use the safety model to define the context in which system 100 can safely perform autonomous control of vehicle 200. To generate this safety model, in some embodiments, processing unit 110 may consider the positions and movements of other vehicles, detected curbs and guardrails, and / or general road shape descriptions extracted from map data (such as data from map database 160). By considering additional information sources, processing unit 110 can provide redundancy for detecting road signs and lane geometry, and increase the reliability of system 100.

[0162] Figure 5D This is a flowchart illustrating an exemplary process 500D for detecting traffic lights in a set of images, consistent with the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500D. At step 560, processing unit 110 may scan the set of images and identify objects appearing in the images that may contain locations of traffic lights. For example, processing unit 110 may filter the identified objects to construct a set of candidate objects, excluding those that are unlikely to correspond to traffic lights. Filtering may be done based on various attributes associated with traffic lights, such as shape, size, texture, location (e.g., relative to vehicle 200), etc. Such attributes may be based on multiple examples of traffic lights and traffic control signals and stored in a database. In some embodiments, processing unit 110 may perform multi-frame analysis on the set of candidate objects reflecting possible traffic lights. For example, processing unit 110 may track candidate objects across consecutive image frames, estimate the real-world location of the candidate objects, and filter out moving objects (which are unlikely to be traffic lights). In some embodiments, the processing unit 110 may perform color analysis on candidate objects and identify the relative positions of detected colors appearing in possible traffic lights.

[0163] At step 562, processing unit 110 can analyze the geometry of the intersection. The analysis can be based on any combination of: (i) the number of lanes detected on either side of vehicle 200, (ii) signs detected on the road (e.g., arrow signs), and (iii) a description of the intersection extracted from map data (e.g., data from map database 160). Processing unit 110 can use information derived from the monocular analysis module 402 for the analysis. Furthermore, processing unit 110 can determine the correspondence between traffic lights detected in step 560 and lanes appearing near vehicle 200.

[0164] At step 564, as vehicle 200 approaches the intersection, processing unit 110 can update the confidence level associated with the analyzed intersection geometry and detected traffic lights. For example, comparing the estimated number of traffic lights present at the intersection with the actual number present can affect the confidence level. Therefore, based on this confidence level, processing unit 110 can delegate control to the driver of vehicle 200 to improve safety conditions. By performing steps 560, 562, and 564, processing unit 110 can identify traffic lights appearing in the set of captured images and analyze intersection geometry information. Based on this identification and analysis, processing unit 110 can induce one or more navigation responses in vehicle 200, as described above. Figure 5A As described.

[0165] Figure 5E This is a flowchart illustrating an exemplary process 500E for inducing one or more navigation responses in vehicle 200 based on a vehicle path, consistent with the disclosed embodiments. At step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. The vehicle path can be represented using a set of points expressed in coordinates (x, z), and the distance d between any two points in this set of points is... i The distance can fall within a range of 1 to 5 meters. In one embodiment, processing unit 110 can use two polynomials, such as a left-road polynomial and a right-road polynomial, to construct an initial vehicle path. Processing unit 110 can calculate the geometric midpoint between the two polynomials and offset each point included in the resulting vehicle path by a predetermined offset (e.g., intelligent lane offset), if any (zero offset could correspond to driving in the middle of the lane). This offset can be in a direction perpendicular to the road segment between any two points in the vehicle path. In another embodiment, processing unit 110 can use a polynomial and an estimated lane width to offset each point in the vehicle path by half the estimated lane width plus a predetermined offset (e.g., intelligent lane offset).

[0166] At step 572, processing unit 110 may update the vehicle path constructed in step 570. Processing unit 110 may use a higher resolution to reconstruct the vehicle path constructed in step 570, such that the distance between two points in the set representing the vehicle path is d. k Less than the distance d mentioned above i For example, the distance dk can fall within the range of 0.1 to 0.3 meters. The processing unit 110 can reconstruct the vehicle path using a parabolic spline algorithm, which can produce a cumulative distance vector S corresponding to the total length of the vehicle path (i.e., based on the set of points representing the vehicle path).

[0167] At step 574, processing unit 110 can determine the look-ahead point (expressed in coordinates as (x, y, y) based on the updated vehicle path constructed at step 572. l , z l Processing unit 110 can extract a forward-looking point from the accumulated distance vector S, and this forward-looking point can be associated with a forward-looking distance and a forward-looking time. The forward-looking distance can have a lower limit ranging from 10 meters to 20 meters and can be calculated as the product of the vehicle 200's speed and the forward-looking time. For example, as the speed of the vehicle 200 decreases, the forward-looking distance can also decrease (e.g., until it reaches the lower limit). The forward-looking time can range from 0.5 to 1.5 seconds and can be inversely proportional to the gain of one or more control loops, such as a heading error tracking control loop, that cause navigation responses in the vehicle 200. For example, the gain of this heading error tracking control loop can depend on the bandwidth of the yaw rate loop, the steering actuator loop, the vehicle's lateral dynamics, etc. Therefore, the higher the gain of the heading error tracking control loop, the shorter the forward-looking time.

[0168] At step 576, processing unit 110 can determine the heading error and yaw rate commands based on the foresight point determined in step 574. Processing unit 110 can do this by calculating the arctangent of the foresight point, for example, arctan(x... l / z l The yaw rate command is determined by the product of the heading error and the high-level control gain. If the look-ahead distance is not at the lower limit, the high-level control gain can be equal to: (2 / look-ahead time). Otherwise, the high-level control gain can be equal to: (2 × vehicle speed 200 / look-ahead distance).

[0169] Figure 5F This is a flowchart illustrating an exemplary process 500F for determining whether a vehicle ahead is changing lanes, consistent with the disclosed embodiments. At step 580, processing unit 110 may determine navigation information associated with the vehicle ahead (e.g., a vehicle traveling in front of vehicle 200). For example, processing unit 110 may use the above combination... Figure 5A and Figure 5B The described technology determines the position, speed (e.g., direction and velocity), and / or acceleration of a vehicle ahead. Processing unit 110 can also be configured to use the above combination... Figure 5E The described technique determines one or more road polynomials, forward viewpoints (associated with vehicle 200), and / or trajectories (e.g., a set of points describing the path taken by the vehicle ahead).

[0170] At step 582, processing unit 110 may analyze the navigation information determined in step 580. In one embodiment, processing unit 110 may calculate the distance between the tracking trajectory and the road polynomial (e.g., along the trajectory). If the change in this distance along the trajectory exceeds a predetermined threshold (e.g., 0.1 to 0.2 meters on a straight road, 0.3 to 0.4 meters on a moderately curved road, and 0.5 to 0.6 meters on a sharp turn), processing unit 110 may determine that the vehicle ahead is likely changing lanes. In the case where multiple vehicles are detected traveling in front of vehicle 200, processing unit 110 may compare the tracking trajectory associated with each vehicle. Based on this comparison, processing unit 110 may determine that a vehicle whose tracking trajectory does not match the tracking trajectories of other vehicles is likely changing lanes. Processing unit 110 may additionally compare the curvature of the tracking trajectory (associated with the vehicle ahead) with the expected curvature of the road segment in which the vehicle ahead is traveling. The desired curvature can be extracted from map data (e.g., data from map database 160), from road polynomials, from tracking trajectories of other vehicles, from existing knowledge about the road, etc. If the difference between the curvature of the tracking trajectory and the desired curvature of the road segment exceeds a predetermined threshold, the processing unit 110 can determine that the vehicle ahead is likely changing lanes.

[0171] In another embodiment, processing unit 110 may compare the instantaneous position of the vehicle ahead with a forward viewpoint (associated with vehicle 200) over a specific time period (e.g., 0.5 to 1.5 seconds). If the distance between the instantaneous position of the vehicle ahead and the forward viewpoint changes during this specific time period, and the cumulative sum of the changes exceeds a predetermined threshold (e.g., 0.3 to 0.4 meters on a straight road, 0.7 to 0.8 meters on a moderately curved road, and 1.3 to 1.7 meters on a sharp turn), then processing unit 110 may determine that the vehicle ahead is likely changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the tracking trajectory by comparing the lateral distance traveled along the tracking trajectory with the desired curvature of the tracking path. The desired radius of curvature can be determined by calculation: (δ... z 2 +δ x 2 ) / 2 / (δ x ), where δ x Indicates the lateral travel distance and δ zThe longitudinal distance traveled is represented. If the difference between the lateral distance traveled and the desired curvature exceeds a predetermined threshold (e.g., 500 to 700 meters), the processing unit 110 can determine that the vehicle ahead is likely changing lanes. In another embodiment, the processing unit 110 can analyze the position of the vehicle ahead. If the position of the vehicle ahead obscures the road polynomial (e.g., the vehicle ahead is overlaid on the road polynomial), the processing unit 110 can determine that the vehicle ahead is likely changing lanes. If the position of the vehicle ahead is such that another vehicle is detected in front of the vehicle ahead and the tracking trajectories of the two vehicles are not parallel, the processing unit 110 can determine that the (closer) vehicle ahead is likely changing lanes.

[0172] In step 584, processing unit 110 may determine whether the vehicle 200 ahead is changing lanes based on the analysis performed in step 582. For example, processing unit 110 may make this determination based on a weighted average of the various analyses performed in step 582. In such a scheme, for example, a determination by processing unit 110 based on a particular type of analysis that the vehicle ahead is likely to change lanes may be assigned a value "1" (and "0" to indicate a determination that the vehicle ahead is unlikely to change lanes). Different analyses performed at step 582 may be assigned different weights, and the disclosed embodiments are not limited to any particular combination of analysis and weights.

[0173] Figure 6 This is a flowchart illustrating an exemplary process 600 for evoking one or more navigation responses based on stereoscopic image analysis, consistent with the disclosed embodiments. At step 610, processing unit 110 may receive first and second plurality of images via data interface 128. For example, a camera included in image acquisition unit 120 (such as image capture devices 122 and 124 having fields of view 202 and 204) may capture first and second plurality of images of an area in front of vehicle 200 and transmit them to processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive the first and second plurality of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0174] At step 620, the processing unit 110 can execute the stereo image analysis module 404 to perform stereo image analysis on the first and second plurality of images to create a 3D map of the road in front of the vehicle and detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. This can be combined with the above. Figures 5A to 5DStereo image analysis is performed in a similar manner to the described steps. For example, processing unit 110 may execute stereo image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road signs, traffic lights, road hazards, etc.) in first and second plurality of images, filter out subsets of candidate objects based on various criteria, and perform multi-frame analysis, construct measurements, and determine confidence levels for the remaining candidate objects. In performing the above steps, processing unit 110 may consider information from both the first and second plurality of images, rather than just information from one set of images. For example, processing unit 110 may analyze differences in pixel-level data (or other subsets of data from the two streams of captured images) of candidate objects appearing in both the first and second plurality of images. As another example, processing unit 110 may estimate the position and / or velocity of a candidate object (e.g., relative to vehicle 200) by observing that an object appears in one of the plurality of images but not in another, or other differences that may exist relative to objects appearing in the two image streams. For example, the position, velocity, and / or acceleration relative to vehicle 200 may be determined based on features such as trajectory, position, and movement characteristics associated with an object appearing in one or both of the image streams.

[0175] At step 630, processing unit 110 may execute navigation response module 408 to perform the analysis performed in step 620 and as described above. Figure 4 The described technology induces one or more navigation responses in vehicle 200. Navigation responses may include, for example, steering, lane changing, changes in acceleration, changes in speed, braking, etc. In some embodiments, processing unit 110 may use data derived from execution speed and acceleration module 406 to induce the one or more navigation responses. Furthermore, multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof.

[0176] Figure 7This is a flowchart illustrating an exemplary process 700 consistent with the disclosed embodiments for evoking one or more navigation responses based on the analysis of three sets of images. At step 710, processing unit 110 may receive first, second, and third plurality of images via data interface 128. For example, cameras included in image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may capture first, second, and third plurality of images of areas in front of and / or to the sides of vehicle 200 and transmit them to processing unit 110 via digital connections (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, processing unit 110 may receive first, second, and third plurality of images via three or more data interfaces. For example, each of image capture devices 122, 124, and 126 may have an associated data interface for communicating data to processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0177] In step 720, processing unit 110 can analyze the first, second, and third images to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, and road hazards. This analysis can be performed in a manner similar to the combination described above. Figures 5A-5D and Figure 6 The steps described are performed in a manner that allows for such analysis. For example, processing unit 110 can perform monocular image analysis on each of the first, second, and third plurality of images (e.g., via execution by monocular image analysis module 402 and based on the above combination). Figures 5A to 5D The steps described. Alternatively, processing unit 110 may perform stereoscopic image analysis on the first and second plurality of images, the second and third plurality of images, and / or the first and third plurality of images (e.g., via execution by stereoscopic image analysis module 404 and based on the above combination). Figure 6 (The steps described). Processed information corresponding to the analysis of the first, second, and / or third plurality of images can be combined. In some embodiments, processing unit 110 can perform a combination of monocular and stereoscopic image analysis. For example, processing unit 110 can perform monocular image analysis on the first plurality of images (e.g., via execution of monocular image analysis module 402) and stereoscopic image analysis on the second and third plurality of images (e.g., via execution of stereoscopic image analysis module 404). The configuration of image capture devices 122, 124, and 126—including their respective positioning and fields of view 202, 204, and 206—can affect the type of analysis performed on the first, second, and third plurality of images. The disclosed embodiments are not limited to a specific configuration of image capture devices 122, 124, and 126 or the type of analysis performed on the first, second, and third plurality of images.

[0178] In some embodiments, processing unit 110 may test system 100 based on the images acquired and analyzed in steps 710 and 720. Such testing can provide an indicator of the overall performance of system 100 for certain configurations of image acquisition devices 122, 124, and 126. For example, processing unit 110 may determine the proportion of “false hits” (e.g., situations where system 100 incorrectly determines the presence of vehicles or pedestrians) and “missed hits”.

[0179] At step 730, processing unit 110 may induce one or more navigation responses in vehicle 200 based on information derived from two of the first, second, and third plurality of images. The selection of two of the first, second, and third plurality of images may depend on various factors, such as, for example, the number, type, and size of objects detected in each of the plurality of images. Processing unit 110 may also make the selection based on image quality and resolution, the effective field of view reflected in the image, the number of captured frames, the extent to which one or more objects of interest actually appear in the frames (e.g., the percentage of frames in which the object appears, the proportion of objects appearing in each such frame), etc.

[0180] In some embodiments, processing unit 110 can select two pieces of information derived from a first, second, and third plurality of images by determining the degree of consistency between information derived from one image source and information derived from other image sources. For example, processing unit 110 can combine the processed information derived from each of the image capture devices 122, 124, and 126 (whether through monocular analysis, stereo analysis, or any combination of both) and determine visual indicators (e.g., lane markings, detected vehicles and their locations and / or paths, detected traffic lights, etc.) that are consistent between each captured image from the image capture devices 122, 124, and 126. Processing unit 110 can also exclude inconsistent information between the captured images (e.g., vehicles changing lanes, lane models indicating that a vehicle is too close to vehicle 200, etc.). Therefore, processing unit 110 can select two pieces of information derived from the first, second, and third plurality of images based on determining consistent and inconsistent information.

[0181] The navigation response may include, for example, changes in steering, lane changing, braking, and acceleration. The processing unit 110 may base its response on the analysis performed in step 720 and the above-mentioned combination. Figure 4The described techniques are used to induce one or more navigation responses. Processing unit 110 may also use data derived from execution speed and acceleration module 406 to induce one or more navigation responses. In some embodiments, processing unit 110 may induce one or more navigation responses based on the relative position, relative velocity, and / or relative acceleration between the vehicle 200 and objects detected within any of the first, second, and third plurality of images. The multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof.

[0182] Sparse road model for autonomous vehicle navigation

[0183] In some embodiments, the disclosed systems and methods may use sparse maps for autonomous vehicle navigation. Specifically, sparse maps can be used for autonomous vehicle navigation along road segments. For example, sparse maps can provide sufficient information for navigating an autonomous vehicle without storing and / or updating large amounts of data. As discussed in more detail below, an autonomous vehicle may use sparse maps based on one or more stored trajectories to navigate one or more roads.

[0184] Sparse maps for autonomous vehicle navigation

[0185] In some embodiments, the disclosed systems and methods can generate sparse maps for autonomous vehicle navigation. For example, sparse maps can provide sufficient information for navigation without requiring excessive data storage or data transfer rates. As discussed in more detail below, a vehicle (which may be an autonomous vehicle) can use a sparse map to navigate one or more roads. For example, in some embodiments, the sparse map may contain data related to roads and potential landmarks along those roads, which may be sufficient for vehicle navigation but also exhibit a small data footprint. For example, the sparse data map described in detail below may require significantly less storage space and data transfer bandwidth compared to a digital map containing detailed map information, such as image data collected along roads.

[0186] For example, sparse data maps can store a three-dimensional polynomial representation of preferred vehicle routes along a road, rather than a detailed representation of road segments. These routes may require very little data storage space. Furthermore, in the described sparse data maps, landmarks can be identified and included in the sparse map road model to aid navigation. These landmarks can be located at any spacing suitable for enabling vehicle navigation, but in some cases, they do not need to be identified and included in the model at high density and short spacing. Instead, in some cases, navigation based on landmarks spaced at least 50 meters, 100 meters, 500 meters, 1 kilometer, or 2 kilometers apart is possible. As will be discussed in more detail in other sections, sparse maps can be generated based on data collected or measured by vehicles equipped with various sensors and devices (such as image capture devices, GPS sensors, motion sensors, etc.) as the vehicles travel along the road. In some cases, sparse maps can be generated based on data collected during multiple drives of one or more vehicles along a particular road. Generating sparse maps using multiple drives of one or more vehicles can be referred to as “crowdsourced” sparse maps.

[0187] Consistent with the disclosed embodiments, autonomous vehicle systems can use sparse maps for navigation. For example, the disclosed systems and methods can allocate sparse maps to generate road navigation models for autonomous vehicles, and the sparse maps and / or the generated road navigation models can be used to navigate autonomous vehicles along road segments. Sparse maps consistent with this disclosure can contain one or more three-dimensional contours that can represent predetermined traverses that autonomous vehicles can traverse as they move along associated road segments.

[0188] Sparse maps consistent with this disclosure may also contain data representing one or more road features. Such road features may include identifiable landmarks, road signature outlines, and any other road-related features useful in vehicle navigation. Sparse maps consistent with this disclosure can enable autonomous vehicle navigation based on a relatively small amount of data contained in the sparse map. For example, the disclosed embodiments of sparse maps may require relatively little storage space (and relatively little bandwidth when portions of the sparse map are transmitted to the vehicle), but can still adequately provide autonomous vehicle navigation, rather than containing detailed representations of roads, such as curbs, road curvature, images associated with road segments, or data describing other physical features associated with road segments. In some embodiments, the small data footprint of the disclosed sparse map can be achieved by storing representations of road-related elements that require a small amount of data but still enable autonomous navigation, which will be discussed in further detail below.

[0189] For example, the disclosed sparse map can store a polynomial representation of one or more trajectories that a vehicle can follow along a road, rather than storing detailed representations of all aspects of the road. Therefore, using the disclosed sparse map, vehicles can be navigated along specific road segments instead of storing (or having to transmit) details about the physical properties of the road to enable navigation along it. In some cases, it is not necessary to interpret the physical aspects of the road, but rather to align the path the vehicle travels with a trajectory (e.g., a polynomial spline) along the specific road segment. In this way, vehicles can be navigated primarily based on the stored trajectory (e.g., a polynomial spline), which can require significantly less storage space compared to methods involving the storage of road images, road parameters, road layouts, etc.

[0190] In addition to the stored polynomial representation of the trajectory along the road segment, the disclosed sparse map may also contain small data objects that can represent road features. In some embodiments, the small data objects may contain digital signatures derived from digital images (or digital signals) acquired by sensors (e.g., cameras or other sensors, such as suspension sensors) on a vehicle traveling along the road segment. The digital signatures may have a reduced size relative to the signals acquired by the sensors. In some embodiments, the digital signatures may be created to be compatible with a classifier function configured to, for example, detect and identify road features from signals acquired by sensors during subsequent driving. In some embodiments, digital signatures may be created such that they have the smallest possible footprint while preserving the ability to associate or match road features with images based on road features (or, if the stored signature is not based on images and / or contains other data, digital signals generated by sensors), images of which are captured by cameras on vehicles subsequently traveling along the same road segment.

[0191] In some embodiments, the size of the data object may be further correlated with the uniqueness of the road feature. For example, for a road feature that can be detected by a camera on a vehicle, and wherein the camera system on the vehicle is coupled to a classifier that is able to classify image data corresponding to the road feature as being associated with a specific type of road feature (e.g., a road sign), and wherein such a road sign is locally unique in the area (e.g., there are no identical road signs or road signs of the same type nearby), storing data indicating the type of the road feature and its location may be sufficient.

[0192] As will be discussed in further detail below, road features (e.g., landmarks along road segments) can be stored as small data objects that can represent road features in relatively few bytes while providing sufficient information for identification and navigation using such features. In one example, road signs can be identified as landmarks upon which vehicle navigation can be based. The representation of road signs can be stored in a sparse map to include, for example, a few bytes of data indicating the type of landmark (e.g., a stop sign) and a few bytes of data indicating the location of the landmark (e.g., coordinates). Navigation based on such a data-light representation of landmarks (e.g., using a representation sufficient for landmark-based positioning, identification, and navigation) can provide the desired level of navigation capabilities associated with a sparse map without significantly increasing the data overhead associated with it. This concise representation of landmarks (and other road features) can leverage sensors and processors included on such vehicles, configured to detect, identify, and / or classify specific road features.

[0193] For example, when a sign or even a sign of a particular type in a given area is locally unique (e.g., when there are no other signs or other signs of the same type), a sparse map can use data indicating the type of landmark (sign or sign of a particular type), and during navigation (e.g., autonomous navigation), when a camera on an autonomous vehicle captures an image of an area containing a sign (or sign of a particular type), the processor can process the image, detect the sign (if it is indeed present in the image), classify the image as a sign (or sign of a particular type), and associate the location of the image with the location of the sign stored in the sparse map.

[0194] Generating sparse maps

[0195] In some embodiments, a sparse map may include at least one line representation of road surface features extending along a road segment and multiple landmarks associated with that road segment. In some aspects, sparse maps may be generated via “crowdsourcing,” for example, through image analysis of multiple images acquired as one or more vehicles traverse the road segment.

[0196] Figure 8A sparse map 800 is shown that can be accessed by one or more vehicles (e.g., vehicle 200, which may be an autonomous vehicle) for providing autonomous vehicle navigation. The sparse map 800 may be stored in memory, such as memory 140 or 150. Such a memory device may comprise any type of non-transitory storage device or computer-readable medium. For example, in some embodiments, memory 140 or 150 may comprise a hard disk drive, optical disk, flash memory, magnetic-based memory device, optical-based memory device, etc. In some embodiments, the sparse map 800 may be stored in a database (e.g., map database 160), which may be stored in memory 140 or 150 or other types of storage devices.

[0197] In some embodiments, the sparse map 800 may be stored on a storage device provided on the vehicle 200 or on a non-transitory computer-readable medium (e.g., storage device included in a navigation system on the vehicle 200). A processor provided on the vehicle 200 (e.g., processing unit 110) may access the sparse map 800 stored on the storage device or computer-readable medium provided on the vehicle 200 to generate navigation instructions for guiding the autonomous vehicle 200 as the vehicle traverses road segments.

[0198] However, the sparse map 800 does not need to be stored locally relative to the vehicle. In some embodiments, the sparse map 800 may be stored on a storage device or computer-readable medium provided on a remote server communicating with the vehicle 200 or a device associated with the vehicle 200. A processor (e.g., processing unit 110) provided on the vehicle 200 may receive data contained in the sparse map 800 from the remote server and may execute data for guiding autonomous driving of the vehicle 200. In such embodiments, the remote server may store all or only a portion of the sparse map 800. Accordingly, the storage device or computer-readable medium provided on the vehicle 200 and / or on one or more additional vehicles may store the remaining portions of the sparse map(s) 800.

[0199] Furthermore, in such embodiments, the sparse map 800 can be accessed by multiple vehicles (e.g., dozens, hundreds, thousands, or millions of vehicles) traversing different road segments. It should also be noted that the sparse map 800 can contain multiple sub-maps. For example, in some embodiments, the sparse map 800 can contain hundreds, thousands, millions, or more sub-maps available for navigation vehicles. Such sub-maps can be referred to as local maps, and vehicles traveling along the road can access any number of local maps related to the vehicle's location. The local map portions of the sparse map 800 can be stored along with Global Navigation Satellite System (GNSS) keys as an index to the database of the sparse map 800. Therefore, while the calculation of the steering angle used for navigating the primary vehicle in this system can be performed without relying on the primary vehicle's GNSS position, road features, or landmarks, this GNSS information can be used to retrieve the relevant local maps.

[0200] Generally, sparse map 800 can be generated based on data collected from one or more vehicles as they travel along a road. For example, using sensors on one or more vehicles (e.g., cameras, speedometers, GPS, accelerometers, etc.), the trajectories of one or more vehicles traveling along a road can be recorded, and a polynomial representation of the preferred trajectory for subsequent journeys along the road can be determined based on the collected trajectories traveled by one or more vehicles. Similarly, data collected by one or more vehicles can help identify potential landmarks along a particular road. Data collected from passing vehicles can also be used to identify road contour information, such as road width contours, road roughness contours, traffic line spacing contours, road conditions, etc. Using the collected information, sparse map 800 can be generated and distributed (e.g., for local storage or via in-flight data transmission) for navigating one or more autonomous vehicles. However, in some embodiments, map generation may not end at the initial generation of the map. As will be discussed in more detail below, sparse map 800 can be continuously or periodically updated based on data collected from vehicles as they continue to traverse the roads contained in sparse map 800.

[0201] The data recorded in the sparse map 800 may include location information based on Global Positioning System (GPS) data. For example, location information may be included in the sparse map 800 for use with various map elements, including, for example, landmark positioning, road outline positioning, etc. The positioning of map elements included in the sparse map 800 can be obtained using GPS data collected from vehicles crossing roads. For example, a vehicle passing an identified landmark can determine the positioning of the identified landmark using GPS location information associated with the vehicle and a determination of the landmark's positioning relative to the vehicle (e.g., image analysis based on data collected from one or more cameras on the vehicle). This positioning determination of the identified landmark (or any other feature included in the sparse map 800) can be repeated as additional vehicles pass the location of the identified landmark. Some or all of the additional positioning determinations can be used to refine the positioning information stored in the sparse map 800 relative to the identified landmarks. For example, in some embodiments, multiple location measurements relative to a specific feature stored in the sparse map 800 may be averaged together. However, any other mathematical operations may also be used to refine the stored positioning of map elements based on multiple determined positioning of the map elements.

[0202] The sparse maps of the disclosed embodiments can enable autonomous navigation of vehicles using relatively little stored data. In some embodiments, the sparse map 800 may have a data density of less than 2 MB per kilometer of road, less than 1 MB per kilometer of road, less than 500 kB per kilometer of road, or less than 100 kB per kilometer of road (e.g., containing data representing target trajectories, landmarks, and any other stored road features). In some embodiments, the data density of the sparse map 800 may be less than 10 kB per kilometer of road, or even less than 2 kB per kilometer of road (e.g., 1.6 kB per kilometer), or no more than 10 kB per kilometer of road, or no more than 20 kB per kilometer of road. In some embodiments, a sparse map with a total of 4 GB or less of data may also be used from most (if not all) roads in the primary navigation United States. These data density values ​​may represent averages over the entire sparse map 800, local maps within the sparse map 800, and / or specific road segments within the sparse map 800.

[0203] As described above, the sparse map 800 may contain representations 810 of multiple target trajectories for guiding autonomous driving or navigation along road segments. These target trajectories may be stored as 3D splines. For example, target trajectories stored in the sparse map 800 may be determined based on two or more reconstructed trajectories of a vehicle's previous traversal along a particular road segment. Road segments may be associated with a single target trajectory or multiple target trajectories. For example, on a two-lane road, a first target trajectory may be stored to represent an intended path of travel along the road in a first direction, and a second target trajectory may be stored to represent an intended path of travel along the road in another direction (e.g., opposite to the first direction). Additional target trajectories may be stored for a particular road segment. For example, on a multi-lane road, one or more target trajectories may be stored representing the intended path of travel for vehicles in one or more lanes associated with the multi-lane road. In some embodiments, each lane of the multi-lane road may be associated with its own target trajectory. In other embodiments, the number of target trajectories stored may be less than the number of lanes present on the multi-lane road. In this scenario, a vehicle navigating on a multi-lane road can use any stored target trajectory to guide its navigation by taking into account lane offsets from the lanes on the stored target trajectory (e.g., if a vehicle is traveling in the leftmost lane of a three-lane highway and the target trajectory is stored only for the middle lane of the highway, when navigation instructions are generated, the vehicle can use the target trajectory of the middle lane to navigate by taking into account the lane offsets between the middle lane and the leftmost lane).

[0204] In some embodiments, the target trajectory may represent an ideal path that the vehicle should take while traveling. The target trajectory may be located approximately at the center of, for example, the driving lane. In other cases, the target trajectory may be located elsewhere relative to a road segment. For example, the target trajectory may approximately coincide with the center of the road, the edge of the road, or the edge of a lane. In this case, navigation based on the target trajectory may include a determined offset of the positioning maintenance relative to the target trajectory. Furthermore, in some embodiments, the determined offset of the positioning maintenance relative to the target trajectory may vary based on the type of vehicle (e.g., a two-axle bus may have a different offset along at least a portion of the target trajectory than a truck with more than two axles).

[0205] The sparse map 800 may also contain data associated with multiple predetermined landmarks 820, which are linked to specific road segments, local maps, etc. As discussed in more detail below, these landmarks can be used for navigation of autonomous vehicles. For example, in some embodiments, the landmarks can be used to determine the vehicle's current position relative to a stored target trajectory. Using this position information, the autonomous vehicle can adjust its heading to match the direction of the target trajectory at the determined location.

[0206] Multiple landmarks 820 can be identified and stored in the sparse map 800 at any suitable spacing. In some embodiments, landmarks can be stored at a relatively high density (e.g., every few meters or more). However, in some embodiments, significantly larger landmark spacing values ​​can be used. For example, in the sparse map 800, identified (or recognized) landmarks can be spaced 10 meters, 20 meters, 50 meters, 100 meters, 1 kilometer, or 2 kilometers apart. In some cases, identified landmarks can be located at distances even exceeding 2 kilometers.

[0207] Between landmarks, and thus between the determination of the vehicle's position relative to a target trajectory, the vehicle can navigate based on dead reckoning, in which the vehicle uses sensors to determine its ego motion and estimate its position relative to the target trajectory. Since errors can accumulate during navigation via dead reckoning, the position determination relative to the target trajectory may become increasingly inaccurate over time. The vehicle can use landmarks appearing in the sparse map 800 (and their known locations) to eliminate errors in position determination caused by dead reckoning. In this way, the identified landmarks included in the sparse map 800 can be used as navigation anchors from which the vehicle's accurate position relative to the target trajectory can be determined. Because a certain amount of error in position determination is acceptable, the identified landmarks do not always need to be available to the autonomous vehicle. Instead, suitable navigation can even be based on landmark spacing of 10 meters, 20 meters, 50 meters, 100 meters, 500 meters, 1 kilometer, 2 kilometers, or more, as described above. In some embodiments, a density of one identified landmark per kilometer of road is sufficient to maintain longitudinal position determination accuracy within 1 meter. Therefore, it is not necessary to store every potential landmark that appears along the road segment in the sparse map 800.

[0208] Furthermore, in some embodiments, lane markings can be used for vehicle positioning during landmark intervals. By using lane markings during landmark intervals, the accumulation of navigation time via dead reckoning can be minimized.

[0209] In addition to target trajectories and identified landmarks, sparse maps 800 can contain information related to a variety of other road features. For example, Figure 9A A representation of a curve along a specific road segment is shown that can be stored in a sparse map 800. In some embodiments, a single lane of the road can be modeled by a three-dimensional polynomial description of the left and right sides of the road. Figure 9A The diagram shows a polynomial representing the left and right sides of a single lane. Regardless of the number of lanes a road can have, the road can be structured in a manner similar to... Figure 9A The representation shown uses polynomials. For example, the left and right sides of a multi-lane road can be represented using a polynomial similar to... Figure 9A The polynomial shown is used to represent this, and the middle lane markings on multi-lane roads (e.g., dashed lines indicating lane boundaries, solid yellow lines indicating boundaries between lanes traveling in different directions, etc.) can also be represented using, for example... Figure 9A The polynomial shown is used to represent this.

[0210] like Figure 9A As shown, lane 900 can be represented using a polynomial (e.g., a first-order, second-order, third-order, or any suitable order polynomial). For illustration, lane 900 is shown as a two-dimensional lane, and the polynomial is shown as a two-dimensional polynomial. Figure 9A As shown, lane 900 includes a left side 910 and a right side 920. In some embodiments, more than one polynomial may be used to represent the location of each side of the road or lane boundary. For example, each of the left side 910 and the right side 920 may be represented by multiple polynomials of any suitable length. In some cases, the polynomials may have a length of approximately 100 m, although other lengths greater or less than 100 m may also be used. Furthermore, the polynomials may overlap each other to facilitate seamless transitions in navigation based on subsequently encountered polynomials as the primary vehicle travels along the road. For example, each of the left side 910 and the right side 920 may be represented by multiple third-order polynomials divided into segments of approximately 100 meters in length (an example of a first predetermined range) and overlapping each other by approximately 50 meters. The polynomials representing the left side 910 and the right side 920 may or may not have the same order. For example, in some embodiments, some polynomials may be second-order polynomials, some may be third-order polynomials, and some may be fourth-order polynomials.

[0211] exist Figure 9A In the example shown, the left side 910 of lane 900 is represented by two sets of third-order polynomials. The first set contains polynomial segments 911, 912, and 913. The second set contains polynomial segments 914, 915, and 916. While these two sets are substantially parallel to each other, they follow the positioning of their respective road edges. Polynomial segments 911, 912, 913, 914, 915, and 916 are approximately 100 meters long and overlap with adjacent segments in the series by approximately 50 meters. However, as mentioned earlier, polynomials of different lengths and different overlaps can also be used. For example, polynomials can have lengths of 500m, 1km, or longer, and the overlap can vary from 0 to 50m, 50m to 100m, or greater than 100m. Furthermore, although... Figure 9A These are shown as polynomials extending in 2D space (e.g., on the surface of paper), but it should be understood that these polynomials can represent curves extending in three dimensions (e.g., containing a height component) to represent elevation changes in road segments in addition to XY curvature. Figure 9AIn the example shown, the right side 920 of lane 900 is further represented by a first group having polynomial segments 921, 922 and 923 and a second group having polynomial segments 924, 925 and 926.

[0212] Returning to the target trajectory on the sparse map 800, Figure 9B The diagram illustrates a three-dimensional polynomial representing the target trajectory of a vehicle traveling along a specific road segment. The target trajectory represents not only the XY path the vehicle should take along the specific road segment, but also the elevation changes the vehicle will experience while traveling along that road segment. Therefore, each target trajectory in the sparse map 800 can be represented by one or more three-dimensional polynomials, such as... Figure 9B The three-dimensional polynomial 950 is shown. The sparse map 800 may contain multiple trajectories (e.g., millions or billions or more, to represent the trajectories of vehicles along individual road segments along roads around the world). In some embodiments, each target trajectory may correspond to a spline connecting segments of the three-dimensional polynomial.

[0213] Regarding the data footprint of the polynomial curves stored in the sparse map 800, in some embodiments, each cubic polynomial can be represented by four parameters, each requiring four bytes of data. A suitable representation can be obtained using a cubic polynomial that requires approximately 192 bytes of data per 100m. For a master vehicle traveling at approximately 100km / hr, this translates to a data usage / transmission requirement of approximately 200kB per hour.

[0214] Sparse maps 800 can describe lane networks using a combination of geometric structure descriptors and metadata. The geometry can be described using polynomials or splines as described above. Metadata can describe the number of lanes, special characteristics (such as shared lanes), and any other possible sparse labels. The total footprint of this indicator may be negligible.

[0215] Accordingly, the sparse map according to embodiments of the present disclosure may include at least one line representation of road surface features extending along a road segment, each line representation representing a path along the road segment substantially corresponding to a road surface feature. In some embodiments, as described above, the at least one line representation of the road surface feature may include splines, polynomial representations, or curves. Furthermore, in some embodiments, the road surface feature may include at least one of curbs or lane markings. Additionally, as discussed below regarding “crowdsourcing,” road surface features can be identified through image analysis of multiple images acquired when one or more vehicles traverse a road segment.

[0216] As previously mentioned, the sparse map 800 can contain multiple predetermined landmarks associated with road segments. Each landmark in the sparse map 800 can be represented and identified using less data than the actual images stored, rather than storing images of the landmark and relying on image recognition analysis, for example, based on captured and stored images. The data representing the landmarks can still contain enough information to describe or identify landmarks along the road. Storing data describing the characteristics of the landmarks, rather than actual images of the landmarks, can reduce the size of the sparse map 800.

[0217] Figure 10 Examples of the types of landmarks that can be represented in sparse map 800 are shown. Landmarks can include any visible and identifiable object along a road segment. Landmarks can be selected such that they are fixed and their location and / or content do not change frequently. Landmarks included in sparse map 800 are useful in determining the location of vehicle 200 relative to a target trajectory as a vehicle traverses a particular road segment. Examples of landmarks can include traffic signs, directional signs, general signs (e.g., rectangular signs), roadside fixtures (e.g., lampposts, reflectors, etc.), and any other suitable categories. In some embodiments, lane markings on the road can also be included as landmarks in sparse map 800.

[0218] Figure 10 Examples of landmarks shown include traffic signs, directional signs, roadside fixtures, and general signs. Traffic signs may include, for example, speed limit signs (e.g., speed limit sign 1000), yield signs (e.g., yield sign 1005), route number signs (e.g., route number sign 1010), traffic light signs (e.g., traffic light sign 1015), and stop signs (e.g., stop sign 1020). Directional signs may include signs containing one or more arrows indicating one or more directions to different places. For example, directional signs may include a highway sign 1025 with arrows to guide vehicles to different roads or places, an exit sign 1030 with arrows to guide vehicles off the road, and so on. Accordingly, at least one of the multiple landmarks may include road signs.

[0219] General signs may not be related to traffic. For example, general signs may include billboards used for advertising, or welcome signs near the border between two countries, states, counties, cities, or towns. Figure 10 The general sign 1040 (“Joe’s Restaurant”) is shown. Although the general sign 1040 can have a rectangular shape, such as... Figure 10 As shown, however, the general mark 1040 can have other shapes, such as square, circle, triangle, etc.

[0220] Landmarks may also include roadside fixtures. Roadside fixtures may not be signs or related to traffic or direction. For example, roadside fixtures may include lampposts (e.g., lamppost 1035), power line poles, traffic light poles, etc.

[0221] Landmarks can also include beacons specifically designed for autonomous vehicle navigation systems. For example, such beacons can comprise freestanding structures placed at predetermined intervals to aid in the navigation of the host vehicle. These beacons can also include visual / graphical information (e.g., icons, badges, barcodes, etc.) added to existing road signs, which can be identified or recognized by vehicles traveling along the road segment. Such beacons can also include electronic components. In such embodiments, electronic beacons (e.g., RFID tags, etc.) can be used to transmit non-visual information to the host vehicle. This information can include, for example, landmark identification and / or landmark positioning information that the host vehicle can use to determine its position along a target trajectory.

[0222] In some embodiments, landmarks included in the sparse map 800 may be represented by data objects of a predetermined size. The data representing a landmark may include any suitable parameters for identifying a particular landmark. For example, in some embodiments, landmarks stored in the sparse map 800 may include parameters such as the physical size of the landmark (e.g., to support estimation of distances to the landmark based on a known size / scale), distance to previous landmarks, lateral offset, height, type code (e.g., landmark type—what type of directional sign, traffic sign, etc.), GPS coordinates (e.g., supporting Global Positioning), and any other suitable parameters. Each parameter may be associated with a data size. For example, 8 bytes of data may be used to store the landmark size. 12 bytes of data may be used to specify the distance to previous landmarks, lateral offset, and height. The type code associated with a landmark, such as a directional sign or traffic sign, may require approximately 2 bytes of data. For general landmarks, a 50-byte data storage device may be used to store an image signature enabling the identification of a general landmark. The landmark GPS location may be associated with a 16-byte data storage device. These data sizes for each parameter are merely examples, and other data sizes may be used.

[0223] Representing landmarks in a sparse map 800 in this way provides a concise scheme for efficiently representing landmarks in a database. In some embodiments, signs can be referred to as semantic signs and non-semantic signs. Semantic signs can include any category of signs with standardized meanings (e.g., speed limit signs, warning signs, directional signs, etc.). Non-semantic signs can include any sign not associated with standardized meanings (e.g., general advertising signs, signs identifying commercial establishments, etc.). For example, each semantic sign can be represented using 38 bytes of data (e.g., 8 bytes for size; 12 bytes for distance to previous landmarks, lateral offset, and height; 2 bytes for type code; and 16 bytes for GPS coordinates). The sparse map 800 can use a labeling system to represent landmark types. In some cases, each traffic sign or directional sign can be associated with its own label, which can be stored in the database as part of the landmark identifier. For example, the database can contain approximately 1,000 different labels to represent various traffic signs and approximately 10,000 different labels to represent directional signs. Of course, any suitable number of labels can be used, and additional labels can be created as needed. In some embodiments, a general purpose sign may be less than about 100 bytes (e.g., about 86 bytes, including 8 bytes for size; 12 bytes for distance to previous landmarks, lateral offset and height; 50 bytes for image signature; and 16 bytes for GPS coordinates).

[0224] Therefore, for semantic road signs that do not require image signatures, even with a relatively high landmark density of approximately one landmark per 50 meters, the data density impact on the sparse map 800 can be approximately 760 bytes per kilometer (e.g., 20 landmarks per kilometer × 38 bytes per landmark = 760 bytes). Even for general-purpose signs that include image signature components, the data density impact is approximately 1.72 kilobytes per kilometer (e.g., 20 landmarks per kilometer × 86 bytes per landmark = 1,720 bytes). For semantic road signs, this equates to approximately 76 kB of data usage per hour for a vehicle traveling at 100 km / hr. For general-purpose signs, this equates to approximately 170 kB of data usage per hour for a vehicle traveling at 100 km / hr.

[0225] In some embodiments, a general rectangular object, such as a rectangular marker, can be represented in the sparse map 800 by no more than 100 bytes of data. The representation of a general rectangular object (e.g., a general marker 1040) in the sparse map 800 may include a condensed image signature (e.g., a condensed image signature 1045) associated with the general rectangular object. This condensed image signature can be used, for example, to aid in the identification of the general marker, such as as a landmark. This condensed image signature (e.g., image information derived from the actual image data representing the object) can avoid the need to store the actual image of the object, or the need to perform comparative image analysis on the actual image to identify the landmark.

[0226] refer to Figure 10 The sparse map 800 may contain or store a compressed image signature 1045 associated with the general sign 1040, rather than an actual image of the general sign 1040. For example, after an image capture device (e.g., image capture device 122, 124, or 126) captures an image of the general sign 1040, a processor (e.g., an image processor 190 or any other processor that can process images, located on or at a distance from the host vehicle) may perform image analysis to extract / create a compressed image signature 1045 containing a unique signature or pattern associated with the general sign 1040. In one embodiment, the compressed image signature 1045 may contain shape, color pattern, brightness pattern, or any other features that can be extracted from the image of the general sign 1040 to describe the general sign 1040.

[0227] For example, in Figure 10 In the compressed image signature 1045, the circles, triangles, and stars can represent areas of different colors. Patterns represented by circles, triangles, and stars can be stored in the sparse map 800, for example, within 50 bytes designated to contain the image signature. It is important to note that the circles, triangles, and stars are not necessarily intended to indicate that these shapes are stored as part of the image signature. Rather, these shapes are intended conceptually to represent identifiable areas with discernible color differences, text areas, graphic shapes, or other variations of characteristics that can be associated with general signage. This compressed image signature can be used to identify landmarks in the form of general signage. For example, the compressed image signature can be used to perform the same or different analyses based on a comparison of the stored compressed image signature with, for example, image data captured using a camera on an autonomous vehicle.

[0228] Accordingly, multiple landmarks can be identified by performing image analysis on multiple images acquired when one or more vehicles traverse a road segment. As explained below regarding “crowdsourcing,” in some embodiments, the image analysis for identifying multiple landmarks may include accepting potential landmarks when the ratio of images in which the landmark actually appears to images in which the landmark does not appear exceeds a threshold. Furthermore, in some embodiments, the image analysis for identifying multiple landmarks may include rejecting potential landmarks when the ratio of images in which the landmark does not appear to images in which the landmark actually appears exceeds a threshold.

[0229] Returning to the main vehicle can be used to navigate a target trajectory on a specific road segment. Figure 11A The diagram illustrates a polynomial representation of a trajectory captured during the construction or maintenance of a sparse map 800. The polynomial representation of a target trajectory included in the sparse map 800 may be determined based on two or more reconstructed trajectories previously traversed by a vehicle along the same road segment. In some embodiments, the polynomial representation of a target trajectory included in the sparse map 800 may be an aggregation of two or more reconstructed trajectories previously traversed by a vehicle along the same road segment. In some embodiments, the polynomial representation of a target trajectory included in the sparse map 800 may be the average of two or more reconstructed trajectories previously traversed by a vehicle along the same road segment. Other mathematical operations may also be used to construct a target trajectory along a road path based on reconstructed trajectories collected from vehicles traversing along the road segment.

[0230] like Figure 11A As shown, road segment 1100 can be traveled by multiple vehicles 200 at different times. Each vehicle 200 can collect data related to the path taken by the vehicle along the road segment. The path traveled by a particular vehicle can be determined based on camera data, accelerometer information, speed sensor information and / or GPS information, as well as other potential sources. This data can be used to reconstruct the trajectory of the vehicle traveling along the road segment, and based on these reconstructed trajectories, a target trajectory (or multiple target trajectories) can be determined for a particular road segment. This target trajectory can represent the preferred path of the primary vehicle as it travels along the road segment (e.g., guided by an autonomous navigation system).

[0231] exist Figure 11AIn the example shown, the first reconstructed trajectory 1101 can be determined based on data received from a first vehicle that traverses road segment 1100 during a first time period (e.g., day 1), the second reconstructed trajectory 1102 can be obtained from a second vehicle that traverses road segment 1100 during a second time period (e.g., day 2), and the third reconstructed trajectory 1103 can be obtained from a third vehicle that traverses road segment 1100 during a third time period (e.g., day 3). Each trajectory 1101, 1102, and 1103 can be represented by a polynomial, such as a three-dimensional polynomial. It should be noted that in some embodiments, any reconstructed trajectory can be assembled from vehicles that traverse road segment 1100.

[0232] Additionally or alternatively, such reconstructed trajectories can be determined on the server side based on information received from vehicles traversing road segment 1100. For example, in some embodiments, vehicles 200 may transmit data related to their movement along road segment 1100 (e.g., steering angle, heading, time, position, speed, sensed road geometry and / or sensed landmarks, etc.) to one or more servers. The server can reconstruct the trajectory of vehicle 200 based on the received data. The server can also generate target trajectories for navigation of autonomous vehicles that will travel along the same road segment 1100 at a later time, based on first trajectory 1101, second trajectory 1102, and third trajectory 1103. While target trajectories can be associated with a single previous crossing of the road segment, in some embodiments, each target trajectory included in the sparse map 800 may be determined based on two or more reconstructed trajectories of vehicles traversing the same road segment. Figure 11A In this context, the target trajectory is represented by 1110. In some embodiments, the target trajectory 1110 may be generated based on the average of the first trajectory 1101, the second trajectory 1102, and the third trajectory 1103. In some embodiments, the target trajectory 1110 included in the sparse map 800 may be an aggregation (e.g., a weighted combination) of two or more reconstructed trajectories.

[0233] Figure 11B and Figure 11C The concept of a target trajectory associated with road segments existing within geographic region 1111 is further illustrated. For example... Figure 11BAs shown, the first road segment 1120 within geographic region 1111 may include a multi-lane road comprising two lanes 1122 designated for vehicles traveling in a first direction and two additional lanes 1124 designated for vehicles traveling in a second direction opposite to the first direction. Lanes 1122 and lanes 1124 may be separated by double yellow lines 1123. Geographic region 1111 may also include branch road segments 1130 intersecting with road segment 1120. Road segment 1130 may include a two-lane road, with each lane designated for a different direction of travel. Geographic region 1111 may also include other road features such as stop lines 1132, stop signs 1134, speed limit signs 1136, and hazard signs 1138.

[0234] like Figure 11C As shown, the sparse map 800 may include a local map 1140 containing a road model to aid autonomous navigation for vehicles within geographic region 1111. For example, the local map 1140 may contain target trajectories for one or more lanes associated with road segments 1120 and / or 1130 within geographic region 1111. For instance, the local map 1140 may contain target trajectories 1141 and / or 1142 that the autonomous vehicle can access or rely on when traversing lane 1122. Similarly, the local map 1140 may contain target trajectories 1143 and / or 1144 that the autonomous vehicle can access or rely on when traversing lane 1124. Furthermore, the local map 1140 may contain target trajectories 1145 and / or 1146 that the autonomous vehicle can access or rely on when traversing road segment 1130. Target trajectory 1147 represents the preferred path that the autonomous vehicle should follow when transitioning from lane 1120 (specifically, relative to target trajectory 1141 associated with the rightmost lane of lane 1120) to road segment 1130 (specifically, relative to target trajectory 1145 associated with the first side of road segment 1130). Similarly, target trajectory 1148 represents the preferred path that the autonomous vehicle should follow when transitioning from road segment 1130 (specifically, relative to target trajectory 1146) to a portion of road segment 1124 (specifically, as shown, relative to target trajectory 1143 associated with the left lane of lane 1124).

[0235] The sparse map 800 may also include representations of other road-related features associated with geographic region 1111. For example, the sparse map 800 may also include representations of one or more landmarks identified in geographic region 1111. These landmarks may include a first landmark 1150 associated with stop line 1132, a second landmark 1152 associated with stop sign 1134, a third landmark associated with speed limit sign 1154, and a fourth landmark 1156 associated with hazard sign 1138. Such landmarks can be used, for example, to help an autonomous vehicle determine its current location relative to any indicated target trajectory, allowing the vehicle to adjust its heading to match the direction of the target trajectory at the determined location.

[0236] In some embodiments, the sparse map 800 may also include road signature profiles. These road signature profiles may be associated with any discernible / measurable variation in at least one parameter associated with a road. For example, in some cases, such profiles may be associated with variations in road surface information, such as variations in surface roughness of a particular road segment, variations in road width on a particular road segment, variations in the distance between dashed lines drawn along a particular road segment, variations in road curvature along a particular road segment, etc. Figure 11D An example of a road signature profile 1160 is shown. While profile 1160 can represent any of the parameters mentioned above or other parameters, in one example, profile 1160 can represent a measurement of road surface roughness, for example, obtained by monitoring one or more sensors that provide outputs indicating the amount of suspension displacement as the vehicle travels on a particular road segment.

[0237] Alternatively or simultaneously, profile 1160 may represent variations in road width, as determined based on image data obtained via a camera on a vehicle traveling on a particular road segment. For example, such a profile is useful in determining the specific position of an autonomous vehicle relative to a particular target trajectory. That is, as it traverses a road segment, the autonomous vehicle can measure a profile associated with one or more parameters associated with the road segment. If the measured profile can be correlated / matched with a predetermined profile that varies relative to position mapping parameters along the road segment, the measured and predetermined profiles (e.g., by overlaying corresponding portions of the measured and predetermined profiles) can be used to determine the current position along the road segment, and thus the current position relative to a target trajectory along the road segment.

[0238] In some embodiments, the sparse map 800 may contain different trajectories based on different characteristics, environmental conditions, and / or other driving-related parameters associated with the user of the autonomous vehicle. For example, in some embodiments, different trajectories may be generated based on different user preferences and / or profiles. The sparse map 800 containing such different trajectories may be provided to different autonomous vehicles of different users. For example, some users may prefer to avoid toll roads, while others may prefer to take the shortest or fastest route, regardless of whether there are toll roads on the route. The disclosed system may generate different sparse maps with different trajectories based on such different user preferences or profiles. As another example, some users may prefer to drive in fast-moving lanes, while others may prefer to always remain in the center lane.

[0239] Different trajectories can be generated and included in a sparse map 800 based on various environmental conditions, such as day and night, snow, rain, and fog. Autonomous vehicles traveling under different environmental conditions can be provided with sparse maps 800 generated based on these different environmental conditions. In some embodiments, cameras provided on the autonomous vehicle can detect environmental conditions and can provide this information back to the server that generates and provides the sparse map. For example, the server can generate or update an already generated sparse map 800 to include trajectories that may be more suitable or safer for autonomous driving under the detected environmental conditions. As the autonomous vehicle travels along a road, updates to the sparse map 800 based on environmental conditions can be performed dynamically.

[0240] Other driving-related parameters can also be used as the basis for generating different sparse maps and providing different sparse maps to different autonomous vehicles. For example, turning may be more difficult when an autonomous vehicle is traveling at high speed. Trajectories associated with a specific lane rather than the road can be included in the sparse map 800, so that the autonomous vehicle can remain within a specific lane when following a specific trajectory. When an image captured by a camera on the autonomous vehicle indicates that the vehicle has drifted out of the lane (e.g., crossed a lane marking), an action can be triggered inside the vehicle to bring the vehicle back to the designated lane according to the specific trajectory.

[0241] Crowdsourced sparse map

[0242] In some embodiments, the disclosed systems and methods can generate sparse maps for autonomous vehicle navigation. For example, the disclosed systems and methods can use crowdsourced data to generate sparse maps that one or more autonomous vehicles can use to navigate along a road system. As used herein, “crowdsourcing” means receiving data from various vehicles (e.g., autonomous vehicles) traveling on a road segment at different times, and this data is used to generate and / or update a road model. The model can then be transmitted to vehicles or other vehicles traveling later along the road segment to assist autonomous vehicle navigation. The road model can contain multiple target trajectories that represent the preferred trajectories that autonomous vehicles should follow when traversing the road segment. The target trajectories can be the same as the reconstructed actual trajectories collected from vehicles traversing the road segment, which can be transmitted from the vehicles to a server. In some embodiments, the target trajectories can differ from the actual trajectories previously used by one or more vehicles when traversing the road segment. The target trajectories can be generated based on the actual trajectories (e.g., through averaging or any other suitable operation).

[0243] Vehicle trajectory data that vehicles can upload to the server can correspond to the vehicle's actual reconstructed trajectory, or it can correspond to a recommended trajectory, which can be based on or related to the vehicle's actual reconstructed trajectory, but may differ from the actual reconstructed trajectory. For example, vehicles can modify their actual reconstructed trajectories and submit (e.g., a recommendation) the modified actual trajectory to the server. The road model can use the recommended, modified trajectory as the target trajectory for other vehicles' autonomous navigation.

[0244] In addition to trajectory information, other information potentially used in building sparse data maps 800 can include information related to potential landmark candidates. For example, through information crowdsourcing, disclosed systems and methods can identify potential landmarks in the environment and refine landmark locations. Navigation systems for autonomous vehicles can use these landmarks to determine and / or adjust the vehicle's position along the target trajectory.

[0245] A reconstructed trajectory that can be generated as a vehicle travels along a road can be obtained by any suitable method. In some embodiments, the reconstructed trajectory can be generated by stitching together segments of the vehicle's motion along the road using, for example, self-motion estimation (e.g., three-dimensional translation and three-dimensional rotation of the camera (and thus the vehicle body)). Rotation and translation estimations can be determined based on analysis of images captured by one or more image capture devices and information from other sensors or devices, such as inertial sensors and velocity sensors. For example, the inertial sensor may include an accelerometer or other suitable sensor configured to measure changes in vehicle body translation and / or rotation. The vehicle may include a velocity sensor that measures the vehicle's speed.

[0246] In some embodiments, the ego motion of the camera (and thus the vehicle body) can be estimated based on optical flow analysis of the captured images. Optical flow analysis of the image sequence identifies pixel movement in the image sequence and determines the vehicle's motion based on the identified motion. The ego motion can be integrated over time and along road segments to reconstruct a trajectory associated with the road segments already followed by the vehicle.

[0247] Data collected by multiple vehicles driving along a road segment at different times (e.g., reconstructed trajectories) can be used to construct a road model (e.g., containing target trajectories, etc.) included in a sparse data map 800. The data collected by multiple vehicles driving along a road segment at different times can also be averaged to improve the model's accuracy. In some embodiments, data about road geometry and / or landmarks can be received from multiple vehicles traversing a public road segment at different times. This data received from different vehicles can be combined to generate and / or update the road model.

[0248] The geometry of the reconstructed trajectory (and target trajectory) along a road segment can be represented by a curve in three-dimensional space, which can be a spline connecting three-dimensional polynomials. The reconstructed trajectory curve can be determined by analyzing a video stream or multiple images captured by a camera mounted on the vehicle. In some embodiments, a location is identified in each frame or image a few meters ahead of the vehicle's current position. This location is the position where the vehicle is expected to travel within a predetermined time period. This operation can be repeated frame by frame, while the vehicle can calculate the camera's self-motion (rotation and translation). In each frame or image, the vehicle generates a short-range model of the desired path in a reference frame attached to the camera. The short-range models can be stitched together to obtain a three-dimensional model of the road in a coordinate system, which can be arbitrary or predetermined. The three-dimensional model of the road can then be fitted with splines, which can contain or connect one or more polynomials of appropriate order.

[0249] To summarize the short-range road model at each frame, one or more detection modules can be used. For example, a bottom-up lane detection module can be used. The bottom-up lane detection module can be useful when mapping lane markings on a road. This module can find edges in the image and assemble them to form lane markings. A second module can be used in conjunction with the bottom-up lane detection module. The second module is an end-to-end deep neural network that can be trained to predict the correct short-range path from the input image. In both modules, the road model can be detected in the image coordinate system and transformed into a 3D space that can be virtually attached to the camera.

[0250] Although trajectory reconstruction modeling methods may introduce accumulated errors due to the integration of self-motion over long periods, which may include noise components, such errors are likely insignificant because the resulting model can provide sufficient accuracy for navigation at a local scale. Furthermore, integration errors can be eliminated by using external information sources such as satellite imagery or geodesy. For example, the disclosed systems and methods can use a GNSS receiver to eliminate accumulated errors. However, GNSS positioning signals may not always be available and accurate. The disclosed systems and methods enable steering applications that are weakly dependent on the availability and accuracy of GNSS positioning. In such systems, the use of GNSS signals may be limited. For example, in some embodiments, the disclosed system may use GNSS signals only for database indexing purposes.

[0251] In some embodiments, the range scale (e.g., local scale) associated with autonomous vehicle navigation and steering applications can be approximately 50 meters, 100 meters, 200 meters, 300 meters, etc. Such distances can be used because the geometric road model is primarily used for two purposes: planning the forward trajectory and positioning the vehicle on the road model. In some embodiments, when the control algorithm steers the vehicle based on a target point located 1.3 seconds ahead (or any other time, such as 1.5 seconds, 1.7 seconds, 2 seconds, etc.), the planning task can use a model within a typical range of 40 meters ahead (or any other suitable forward distance, such as 20 meters, 30 meters, 50 meters). The positioning task uses a road model within a typical range of 60 meters behind the vehicle (or any other suitable distance, such as 50 meters, 100 meters, 150 meters, etc.). According to a method referred to as “tail alignment,” described in more detail in another section. The disclosed systems and methods can generate geometric models with sufficient accuracy over a specific range (such as 100 meters) such that the planned trajectory does not deviate from the lane center by more than, for example, 30 centimeters.

[0252] As described above, a 3D road model can be constructed by detecting short segments and stitching them together. This stitching can be enabled by calculating a six-degree-of-freedom motion model using video and / or images captured by cameras, data from inertial sensors reflecting vehicle motion, and the main vehicle's speed signal. The accumulated error may be small enough over a localized scale (e.g., approximately 100 meters). All of this can be accomplished in a single drive on a specific road segment.

[0253] In some embodiments, multiple drives can be used to average the resulting model and further improve its accuracy. The same car can drive the same route multiple times, or multiple cars can transmit their collected model data to a central server. In any case, a matching process can be performed to identify overlapping models and enable averaging to generate the target trajectory. Once convergence criteria are met, the constructed model (e.g., containing the target trajectory) can be used for steering. Subsequent drives can be used for further model improvement and adaptation to infrastructure changes.

[0254] If multiple vehicles connect to a central server, sharing driving experiences (such as sensed data) becomes feasible. Each vehicle client can store a partial copy of a general road model, which can be associated with its current location. The bidirectional update process between the vehicles and the server can be performed by both the vehicles and the server. The small footprint concept discussed above enables the disclosed systems and methods to perform bidirectional updates using very little bandwidth.

[0255] Information associated with potential landmarks can also be identified and forwarded to a central server. For example, the disclosed systems and methods can determine one or more physical attributes of potential landmarks based on one or more images containing the landmark. Physical attributes can include the landmark's physical size (e.g., height, width), distance from the vehicle to the landmark, distance between the landmark and previous landmarks, the landmark's lateral position (e.g., the landmark's position relative to a driving lane), the landmark's GPS coordinates, the landmark's type, textual markings on the landmark, etc. For example, a vehicle can analyze one or more images captured by a camera to detect potential landmarks, such as speed limit signs.

[0256] Vehicles can determine the distance from a landmark based on the analysis of one or more images. In some embodiments, suitable image analysis methods, such as scaling methods and / or optical flow methods, can be used to determine the distance based on the analysis of the landmark's images. In some embodiments, the disclosed systems and methods can be configured to determine the type or classification of potential landmarks. When a vehicle determines that a potential landmark corresponds to a predetermined type or classification stored in a sparse map, it is sufficient for the vehicle to communicate the indication of the landmark's type or classification along with its location to a server. The server can store such indications. At a later time, other vehicles can capture images of the landmark, process the images (e.g., using a classifier), and compare the results of the image processing with the indication of the landmark type stored on the server. Various types of landmarks can exist, and different types of landmarks can be associated with different types of data uploaded to and stored on the server. Different processing on the vehicle can detect landmarks and communicate information about them to the server, and the vehicle's systems can receive landmark data from the server and use the landmark data for landmark identification during autonomous navigation.

[0257] In some embodiments, multiple autonomous vehicles traveling on a road segment can communicate with a server. Vehicles (or clients) can generate curves describing their driving in any coordinate system (e.g., through self-motion integration). Vehicles can detect landmarks and locate them within the same frame. Vehicles can upload curves and landmarks to the server. The server can collect data from the vehicles through multiple drives and generate a unified road model. Or, for example, as referenced below... Figure 19 The server can use uploaded curves and landmarks to generate a sparse map with a uniform road model.

[0258] The server can also assign models to clients (e.g., vehicles). For example, the server can assign a sparse map to one or more vehicles. The server can update the model continuously or periodically as new data is received from the vehicles. For example, the server can process the new data to evaluate whether it contains information that should trigger an update or creation of new data on the server. The server can then assign the updated model or updates to the vehicles to provide autonomous vehicle navigation.

[0259] The server can use one or more criteria to determine whether new data received from a vehicle should trigger an update to the model or the creation of new data. For example, when new data indicates that a previously identified landmark at a specific location no longer exists or has been replaced by another landmark, the server can determine that the new data should trigger an update to the model. As another example, when new data indicates that a road segment is closed, and this has been confirmed by data received from other vehicles, the server can determine that the new data should trigger an update to the model.

[0260] The server can assign an updated model (or an updated portion of a model) to one or more vehicles traveling on a road segment associated with the model update. The server can also assign an updated model to a vehicle that is about to travel on a road segment, or a vehicle whose planned journey includes a road segment associated with the model update. For example, when an autonomous vehicle travels along another road segment before reaching the road segment associated with the update, the server can assign the updated or updated model to the autonomous vehicle before the vehicle reaches the road segment.

[0261] In some embodiments, a remote server can collect trajectories and landmarks from multiple clients (e.g., vehicles traveling along a public road segment). The server can use landmarks to match curves and create an average road model based on the trajectories collected from multiple vehicles. The server can also calculate a graph of the road and the most probable path at each node or connection of the road segment. For example, the remote server can align trajectories to generate a crowdsourced sparse map from the collected trajectories.

[0262] The server can average landmark attributes received from multiple vehicles traveling along a public road segment, such as the distance between one landmark and another (e.g., the previous landmark along the road segment) measured by multiple vehicles, to determine arc length parameters and support positioning and speed calibration for each client vehicle along the path. The server can average the physical dimensions of landmarks measured by multiple vehicles traveling along the public road segment and identifying the same landmark. The averaged physical dimensions can be used to support distance estimation, such as the distance from the vehicle to the landmark. The server can average the lateral position of landmarks (e.g., from the lane the vehicle is traveling in to the landmark's location) measured by multiple vehicles traveling along the public road segment and identifying the same landmark. The averaged lateral portion can be used to support lane assignment. The server can average the GPS coordinates of landmarks measured by multiple vehicles traveling along the same road segment and identifying the same landmark. The averaged GPS coordinates of the landmarks can be used to support global localization or positioning of landmarks in the road model.

[0263] In some embodiments, based on data received from the vehicle, the server can identify model changes, such as construction, detours, new signs, sign removal, etc. When new data is received from the vehicle, the server can update the model continuously, periodically, or instantaneously. The server can assign the updated model or the updated model to the vehicle for use in providing autonomous navigation. For example, as discussed further below, the server can use crowdsourced data to filter out “ghost” landmarks detected by the vehicle.

[0264] In some embodiments, the server can analyze driver interventions during autonomous driving. The server can analyze data received from the vehicle at the time of the intervention and its location, and / or data received prior to the time of the intervention. The server can identify portions of the data that caused or is closely related to the intervention, such as data indicating temporary lane closures or data indicating pedestrians in the road. The server can update the model based on the identified data. For example, the server can modify one or more trajectories stored in the model.

[0265] Figure 12 This is a schematic diagram of a system that uses crowdsourcing to generate sparse maps (and uses crowdsourced sparse maps for assignment and navigation). Figure 12 Road segment 1200, containing one or more lanes, is shown. Multiple vehicles 1205, 1210, 1215, 1220, and 1225 may travel on road segment 1200 at the same time or at different times (although...). Figure 12(These are shown as vehicles appearing on road segment 1200 at the same time). At least one of vehicles 1205, 1210, 1215, 1220, and 1225 can be an autonomous vehicle. For the sake of simplicity in this example, it is assumed that all vehicles 1205, 1210, 1215, 1220, and 1225 are autonomous vehicles.

[0266] Each vehicle may be similar to a vehicle disclosed in other embodiments (e.g., vehicle 200) and may include components or devices included in or associated with vehicles disclosed in other embodiments. Each vehicle may be equipped with an image capture device or camera (e.g., image capture device 122 or camera 122). Each vehicle may communicate with a remote server 1230 via one or more networks (e.g., via cellular networks and / or the Internet, etc.) through a wireless communication path 1235 as shown by the dashed line. Each vehicle may transmit data to and receive data from the server 1230. For example, the server 1230 may collect data from multiple vehicles traveling on road segment 1200 at different times and may process the collected data to generate an autonomous vehicle road navigation model or an update to the model. The server 1230 may transmit the autonomous vehicle road navigation model or an update to the model to the vehicle that transmitted data to the server 1230. The server 1230 may transmit the autonomous vehicle road navigation model or an update to the model to other vehicles traveling on road segment 1200 later.

[0267] When vehicles 1205, 1210, 1215, 1220, and 1225 travel on road segment 1200, navigation information collected (e.g., detected, sensed, or measured) by vehicles 1205, 1210, 1215, 1220, and 1225 can be transmitted to server 1230. In some embodiments, the navigation information may be associated with public road segment 1200. The navigation information may include a trajectory associated with each vehicle 1205, 1210, 1215, 1220, and 1225 as each vehicle travels on road segment 1200. In some embodiments, the trajectory may be reconstructed based on data sensed by various sensors and devices provided on vehicle 1205. For example, the trajectory may be reconstructed based on at least one of accelerometer data, speed data, landmark data, road geometry or contour data, vehicle position data, and self-motion data. In some embodiments, the trajectory may be reconstructed based on data from inertial sensors (such as accelerometers) and the speed of vehicle 1205 sensed by a speed sensor. Furthermore, in some embodiments, the trajectory can be determined based on the ego motion of a sensed camera (e.g., via a processor on each of vehicles 1205, 1210, 1215, 1220, and 1225), which can indicate three-dimensional translation and / or three-dimensional rotation (or rotational motion). The ego motion of the camera (and thus the vehicle body) can be determined by analyzing one or more images captured by the camera.

[0268] In some embodiments, the trajectory of vehicle 1205 may be determined by a processor provided on vehicle 1205 and transmitted to server 1230. In other embodiments, server 1230 may receive data sensed by various sensors and devices provided in vehicle 1205 and determine the trajectory based on the data received from vehicle 1205.

[0269] In some embodiments, navigation information transmitted from vehicles 1205, 1210, 1215, 1220, and 1225 to server 1230 may include data about the road surface, road geometry, or road profile. The geometry of road segment 1200 may include lane structure and / or landmarks. Lane structure may include the total number of lanes in road segment 1200, lane type (e.g., single lane, two lanes, driving lane, overtaking lane, etc.), lane markings, lane width, etc. In some embodiments, navigation information may include lane assignment, such as which lane a vehicle is traveling in among multiple lanes. For example, lane assignment may be associated with the numeric value "3," indicating that the vehicle is traveling from the third lane on the left or right. As another example, lane assignment may be associated with the text value "center lane," indicating that the vehicle is traveling in the center lane.

[0270] Server 1230 may store navigation information on a non-transitory computer-readable medium, such as a hard disk drive, optical disk, magnetic tape, memory, etc. Server 1230 may generate (e.g., via a processor included in server 1230) at least a portion of an autonomous vehicle road navigation model for public road segment 1200 based on navigation information received from multiple vehicles 1205, 1210, 1215, 1220, and 1225, and may store this model as part of a sparse map. Server 1230 may determine a trajectory associated with each lane based on crowdsourced data (e.g., navigation information) received from multiple vehicles (e.g., 1205, 1210, 1215, 1220, and 1225) traveling in lanes of the road segment at different times. Server 1230 may generate an autonomous vehicle road navigation model or a portion of the model (e.g., an updated portion) based on the multiple trajectories determined by the crowdsourced navigation data. Server 1230 may transmit the model or an updated portion of the model to one or more of the autonomous vehicles 1205, 1210, 1215, 1220, and 1225 traveling on road segment 1200, or to any other autonomous vehicle traveling on the road segment at a later time, to update the existing autonomous vehicle road navigation model provided in the vehicle's navigation system. The autonomous vehicle road navigation model can be used by autonomous vehicles for autonomous navigation along public road segment 1200.

[0271] As mentioned above, autonomous vehicle road navigation models can be included in sparse maps (e.g., Figure 8 The sparse map 800 depicted in the image contains a sparse record of data relating to road geometry and / or landmarks along the road. This provides sufficient information for autonomous navigation of autonomous vehicles without requiring excessive data storage. In some embodiments, the autonomous vehicle road navigation model may be stored separately from the sparse map 800, and map data from the sparse map 800 may be used when the model is executed for navigation. In some embodiments, the autonomous vehicle road navigation model may use the map data contained in the sparse map 800 to determine a target trajectory along road segment 1200 for use in guiding autonomous vehicles 1205, 1210, 1215, 1220, and 1225, or other vehicles subsequently traveling along road segment 1200. For example, when the autonomous vehicle road navigation model is executed by a processor in a navigation system contained in vehicle 1205, the model may enable the processor to compare a trajectory determined based on navigation information received from vehicle 1205 with a predetermined trajectory contained in the sparse map 800 to verify and / or correct the current driving route of vehicle 1205.

[0272] In an autonomous vehicle road navigation model, the geometry of road features or target trajectories can be encoded by curves in three-dimensional space. In one embodiment, the curve can be a three-dimensional spline containing one or more connected three-dimensional polynomials. As understood by those skilled in the art, a spline can be a numerical function defined piecewise by a series of polynomials for fitting data. The spline used to fit the three-dimensional geometry data of the road can contain linear splines (first order), quadratic splines (second order), cubic splines (third order), or any other splines (other orders), or combinations thereof. The spline can contain one or more three-dimensional polynomials of different orders connecting (e.g., fitting) the data points of the three-dimensional geometry data of the road. In some embodiments, the autonomous vehicle road navigation model can contain a three-dimensional spline corresponding to the target trajectory along a public road segment (e.g., road segment 1200) or a lane of road segment 1200.

[0273] As described above, the autonomous vehicle road navigation model contained in the sparse map may include other information, such as the identification of at least one landmark along road segment 1200. The landmark may be visible within the field of view of a camera (e.g., camera 122) mounted on each of vehicles 1205, 1210, 1215, 1220, and 1225. In some embodiments, camera 122 may capture an image of the landmark. A processor provided on vehicle 1205 (e.g., processor 180, 190, or processing unit 110) may process the image of the landmark to extract landmark identification information. Landmark identification information, rather than an actual image of the landmark, may be stored in the sparse map 800. Landmark identification information may require significantly less storage space than an actual image. Other sensors or systems (e.g., a GPS system) may also provide some identification information about the landmark (e.g., the location of the landmark). Landmarks may include at least one of traffic signs, arrow signs, lane signs, dashed lane signs, traffic lights, stop lines, directional signs (e.g., highway exit signs with arrows indicating directions, highway signs with arrows pointing in different directions or places), landmark beacons, or lampposts. Landmark beacons are devices (e.g., RFID devices) installed along road segments that transmit or reflect signals to receivers installed on vehicles, such that when a vehicle passes the device, the beacon received by the vehicle and the device's location (e.g., determined from the device's GPS positioning) can be used as landmarks to be included in autonomous vehicle road navigation models and / or sparse maps 800.

[0274] The identification of at least one landmark may include the location of at least one landmark. The location of the landmark may be determined based on location measurements performed using sensor systems (e.g., GPS, inertial-based positioning systems, landmark beacons, etc.) associated with multiple vehicles 1205, 1210, 1215, 1220, and 1225. In some embodiments, the location of the landmark may be determined by averaging location measurements detected, collected, or received by sensor systems on different vehicles 1205, 1210, 1215, 1220, and 1225 through multiple driving operations. For example, vehicles 1205, 1210, 1215, 1220, and 1225 may transmit location measurements to server 1230, which may average the location measurements and use the averaged location measurements as the location of the landmark. The location of the landmark may be continuously refined by measurements received from vehicles during subsequent driving operations.

[0275] The landmark's identifier can include its size. A processor provided on a vehicle (e.g., 1205) can estimate the landmark's physical size based on image analysis. Server 1230 can receive multiple estimates of the same landmark's physical size from different vehicles through different driving. Server 1230 can average the different estimates to arrive at the landmark's physical size and store it in the road model. The physical size estimate can be used to further determine or estimate the distance from the vehicle to the landmark. The distance to the landmark can be estimated based on the vehicle's current speed and the expansion ratio of the landmark's position in the image relative to the camera's expanded focal point. For example, the distance to the landmark can be estimated as Z = V * dt * R / D, where V is the vehicle's speed, R is the distance from the landmark in the image at time t1 to the expanded focal point, and D is the change in distance of the landmark in the image from t1 to t2. dt represents (t2 - t1). For example, the distance to a landmark can be estimated using Z = V * dt * R / D, where V is the vehicle speed, R is the distance between the landmark and the extended focal point in the image, dt is the time interval, and D is the image displacement of the landmark along the epipolar line. Other equations equivalent to the above, such as Z = V * ω / Δω, can be used to estimate the distance to a landmark. Here, V is the vehicle speed, ω is the image length (similar to object width), and Δω is the change in the image length per unit time.

[0276] When the physical size of a landmark is known, the distance to the landmark can be determined based on the following equation: Z = f * W / ω, where f is the focal length, W is the size of the landmark (such as height or width), and ω is the number of pixels the landmark leaves the image. According to the above equation, ΔZ = f * W * Δω / ω can be used. 2The change in distance Z is calculated using +f*ΔW / ω, where ΔW decays to zero through averaging, and Δω is the number of pixels representing the accuracy of the bounding box in the image. The estimated physical size of the landmark can be calculated by averaging multiple observations from the server side. The final error in the distance estimation can be very small. When using the above equation, two error sources may emerge: ΔW and Δω. Their contributions to the distance error are given by ΔZ = f*W*Δω / ω. 2 +f*ΔW / ω is given. However, ΔW decays to zero by averaging; therefore, ΔZ is determined by Δω (e.g., the inaccuracy of the bounding box in the image).

[0277] For landmarks of unknown size, the distance to the landmark can be estimated by tracking feature points on the landmark between consecutive frames. For example, certain features appearing on a speed limit sign can be tracked between two or more image frames. Based on these tracked features, a distance distribution per feature point can be generated. The distance estimate can be extracted from the distance distribution. For example, the most frequently occurring distances in the distance distribution can be used as the distance estimate. As another example, the average of the distance distribution can be used as the distance estimate.

[0278] Figure 13 An example autonomous vehicle road navigation model is shown, represented by multiple three-dimensional splines 1301, 1302 and 1303. Figure 13 The curves 1301, 1302, and 1303 shown are for illustrative purposes only. Each spline may contain one or more three-dimensional polynomials connecting multiple data points 1310. Each polynomial may be a first-order polynomial, a second-order polynomial, a third-order polynomial, or any suitable combination of polynomials with different orders. Each data point 1310 may be associated with navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225. In some embodiments, each data point 1310 may be associated with data related to landmarks (e.g., the size, location, and identification information of landmarks) and / or road signature profiles (e.g., road geometry, road roughness profile, road curvature profile, road width profile). In some embodiments, some data points 1310 may be associated with landmark-related data, while other data points may be associated with road signature profile-related data.

[0279] Figure 14The diagram illustrates raw location data 1410 (e.g., GPS data) received from five separate drivers. A driver may be separated from another driver if a separate vehicle is simultaneously passed by another vehicle at the same time, by the same vehicle at the time of separation, or by another vehicle at the time of separation. To account for errors in the location data 1410 and different positions of vehicles within the same lane (e.g., one vehicle may be closer to the left side of the lane than another), server 1230 may use one or more statistical techniques to generate a map skeleton 1420 to determine whether variations in the raw location data 1410 represent actual deviations or statistical errors. Each path within the skeleton 1420 may link back to the raw data 1410 that formed that path. For example, the path between A and B within the skeleton 1420 links to the raw data 1410 from drivers 2, 3, 4, and 5, but not from driver 1. The skeleton 1420 may not be detailed enough for navigating vehicles (e.g., because, unlike the splines described above, the skeleton 1420 combines drivers from multiple lanes on the same road), but it can provide useful topological information and can be used to define intersections.

[0280] Figure 15 An example is shown where additional details can be generated for a sparse map within a road segment of a map skeleton (e.g., road segment A to road segment B within skeleton 1420). Figure 15 As shown, data (e.g., self-motion data, road sign data, etc.) can be represented as a function of the driving position S (or S1 or S2). Server 1230 can identify landmarks in a sparse map by recognizing unique matches between landmarks 1501, 1503, and 1505 of driving 1510 and landmarks 1507 and 1509 of driving 1520. This matching algorithm can also identify landmarks 1511, 1513, and 1515. However, those skilled in the art will recognize that other matching algorithms can be used. For example, probabilistic optimization can be used instead of unique matching or in combination with unique matching. Server 1230 can align driving vehicles longitudinally to align matching landmarks. For example, server 1230 can select one driving vehicle (e.g., driving 1520) as a reference driving vehicle and then shift and / or elastically stretch (multiple) other driving vehicles (e.g., driving 1510) for alignment.

[0281] Figure 16 An example of aligned landmark data used in sparse maps is shown. Figure 16 In the example, landmark 1610 includes road signs. Figure 16 The examples further depict data from multiple driving records 1601, 1603, 1605, 1607, 1609, 1611, and 1613. In Figure 16In the example, the data from drive 1613 contains a "ghost" landmark, and server 1230 can identify it this way because drives 1601, 1603, 1605, 1607, 1609, and 1611 do not contain any identifiers of landmarks near the landmark identified in drive 1613. Accordingly, server 1230 can accept a potential landmark when the ratio of images where the landmark actually appears to images where the landmark does not appear exceeds a threshold, and / or server 1230 can reject a potential landmark when the ratio of images where the landmark does not appear to images where the landmark does appear exceeds a threshold.

[0282] Figure 17 A system 1700 for generating driving data is described, which can be used for crowdsourced sparse maps. For example... Figure 17 As shown, system 1700 may include camera 1701 and locating device 1703 (e.g., GPS locator). Camera 1701 and locating device 1703 may be mounted on a vehicle (e.g., one of vehicles 1205, 1210, 1215, 1220, and 1225). Camera 1701 may generate multiple types of data, such as self-motion data, traffic sign data, road data, etc. Camera data and locating data may be segmented into driving segments 1705. For example, driving segments 1705 may each have camera data and locating data from driving distances of less than 1 kilometer.

[0283] In some embodiments, system 1700 can remove redundancy in driving segment 1705. For example, if a landmark appears in multiple images from camera 1701, system 1700 can remove redundant data so that driving segment 1705 contains only a copy of the landmark's location and any metadata associated with the landmark. As a further example, if a lane sign appears in multiple images from camera 1701, system 1700 can remove redundant data so that driving segment 1705 contains only a copy of the lane sign's location and any metadata associated with the lane sign.

[0284] System 1700 also includes a server (e.g., server 1230). Server 1230 can receive driving segments 1705 from the vehicle and reassemble driving segments 1705 into a single drive 1707. This arrangement reduces bandwidth requirements when data is transferred between the vehicle and the server, while also allowing the server to store data related to the entire driving process.

[0285] Figure 18 The diagram depicts further configurations for crowdsourced sparse maps. Figure 17 System 1700. For example... Figure 17As shown, system 1700 includes vehicle 1810, which uses, for example, cameras (which generate, for example, self-motion data, traffic sign data, road data, etc.) and positioning devices (such as GPS locators) to capture driving data. Figure 17 As shown, vehicle 1810 segments the collected data into driving segments (in... Figure 18 The data is described as "DS1 1", "DS2 1", and "DSN 1". Then, server 1230 receives the driving segments and reconstructs the driving sequence from the received segments (in...). Figure 18 The text is described as "driving 1".

[0286] like Figure 18 As further described, system 1700 also receives data from additional vehicles. For example, vehicle 1820 also uses, for example, cameras (which generate, for example, self-motion data, traffic sign data, road data, etc.) and positioning devices (such as GPS locators) to capture driving data. Similar to vehicle 1810, vehicle 1820 segments the collected data into driving segments (in... Figure 18 The driving segments are described as "DS1 2", "DS22", and "DSN 2". Then, server 1230 receives the driving segments and reconstructs the driving sequence from them (in...). Figure 18 It is described as "Driving 2" in the text. Any number of additional vehicles can be used. For example, Figure 18 It also includes "Car N", which captures driving data and segments it into driving segments (in... Figure 18 Described as "DS1 N", "DS2 N", "DSN N" in the text, and transmitted to server 1230 for reconstruction into a driving (in Figure 18 (Described as "Driving N" in the text).

[0287] like Figure 18 As shown, server 1230 can use reconstructed driving data (e.g., "driving 1", "driving 2", and "driving N") collected from multiple vehicles (e.g., "car 1" (also labeled as vehicle 1810), "car 2" (also labeled as vehicle 1820), and "car N") to construct a sparse map (described as "map").

[0288] Figure 19 This is a flowchart illustrating an example process 1900 for generating a sparse map for autonomous vehicle navigation along a road segment. Process 1900 can be executed by one or more processing devices included in server 1230.

[0289] Process 1900 may include receiving multiple images acquired as one or more vehicles traverse a road segment (step 1905). Server 1230 may receive images from cameras included in one or more of vehicles 1205, 1210, 1215, 1220, and 1225. For example, as vehicle 1205 travels along road segment 1200, camera 122 may capture one or more images of the environment surrounding vehicle 1205. In some embodiments, server 1230 may also receive stripped-down image data that has had redundancy removed by a processor on vehicle 1205, as referenced above. Figure 17 The subject of discussion.

[0290] Process 1900 may further include recognizing at least one line representation of road surface features extending along a road segment based on multiple images (step 1910). Each line representation may represent a path along a road segment substantially corresponding to the road surface features. For example, server 1230 may analyze environmental images received from camera 122 to identify curbs or lane markings and determine a travel trajectory along road segment 1200 associated with curbs or lane markings. In some embodiments, the trajectory (or line representation) may comprise splines, polynomial representations, or curves. Server 1230 may determine the travel trajectory of vehicle 1205 based on camera ego motion (e.g., three-dimensional translation and / or three-dimensional rotational motion) received at step 1905.

[0291] Process 1900 may also include identifying multiple landmarks associated with a road segment based on multiple images (step 1910). For example, server 1230 may analyze environmental images received from camera 122 to identify one or more landmarks, such as road signs along road segment 1200. Server 1230 may use analysis of multiple images acquired when one or more vehicles cross the road segment to identify landmarks. To enable crowdsourcing, the analysis may include rules regarding the acceptance and rejection of potential landmarks associated with the road segment. For example, the analysis may include accepting a potential landmark when the ratio of images in which the landmark actually appears to images in which the landmark does not appear exceeds a threshold, and / or rejecting a potential landmark when the ratio of images in which the landmark does not appear to images in which the landmark actually appears exceeds a threshold.

[0292] Process 1900 may include other operations or steps performed by server 1230. For example, navigation information may include a target trajectory of a vehicle traveling along a road segment, and process 1900 may include server 1230 clustering vehicle trajectories associated with multiple vehicles traveling on the road segment, and determining the target trajectory based on the clustered vehicle trajectories, as discussed in further detail below. Clustering vehicle trajectories may include server 1230 clustering multiple trajectories associated with vehicles traveling on the road segment into multiple clusters based on at least one of the vehicle's absolute heading or the vehicle's lane assignment. Generating the target trajectory may include trajectories averaged by server 1230. As a further example, process 1900 may include aligning the data received in step 1905. As described above, other processes or steps performed by server 1230 may also be included in process 1900.

[0293] The disclosed systems and methods may include other features. For example, the disclosed systems may use local coordinates instead of global coordinates. For autonomous driving, some systems may present data in world coordinates. For example, longitude and latitude coordinates on the Earth's surface may be used. To use a map for steering, the host vehicle can determine its position and orientation relative to the map. Using a GPS device on the vehicle seems natural to locate the vehicle on the map and to find the rotational transformation between the ontology reference frame and the world reference frame (e.g., north, east, and down). Once the ontology reference frame is aligned with the map reference frame, the desired route can be expressed in the ontology reference frame, and steering commands can be calculated or generated.

[0294] The disclosed systems and methods enable autonomous vehicle navigation (e.g., steering control) with low-footprint models that can be collected by the autonomous vehicle itself without the aid of expensive surveying instruments. To support autonomous navigation (e.g., steering applications), the road model can include a sparse map containing the road geometry, its lane structure, and landmarks that can be used to determine the vehicle's location or position along a trajectory included in the model. As described above, the generation of the sparse map can be performed by a remote server that communicates with and receives data from vehicles traveling on the road. The data can include sensed data, trajectories reconstructed based on the sensed data, and / or recommended trajectories that may represent modified reconstructed trajectories. As described below, the server can transmit the model back to the vehicle or other vehicles later traveling on the road to aid in autonomous navigation.

[0295] Figure 20A block diagram of the server is shown. Server 1230 may include a communication unit 2005, which may include hardware components (e.g., communication control circuitry, switches, and antennas) and software components (e.g., communication protocols, computer code). For example, communication unit 2005 may include at least one network interface. Server 1230 can communicate with vehicles 1205, 1210, 1215, 1220, and 1225 via communication unit 2005. For example, server 1230 can receive navigation information transmitted from vehicles 1205, 1210, 1215, 1220, and 1225 via communication unit 2005. Server 1230 can assign autonomous vehicle road navigation models to one or more autonomous vehicles via communication unit 2005.

[0296] Server 1230 may include at least one non-transitory storage medium 2010, such as a hard disk drive, optical disk, magnetic tape, etc. Storage device 1410 may be configured to store data, such as navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225 and / or an autonomous vehicle road navigation model generated by server 1230 based on the navigation information. Storage device 2010 may be configured to store any other information, such as sparse maps (e.g., referenced above). Figure 8 The sparse map under discussion (800).

[0297] In addition to or in place of storage device 2010, server 1230 may include memory 2015. Memory 2015 may be similar to or different from memory 140 or 150. Memory 2015 may be non-transitory memory, such as flash memory, random access memory, etc. Memory 2015 may be configured to store data, such as computer code or instructions executable by a processor (e.g., processor 2020), map data (e.g., data from sparse map 800), autonomous vehicle road navigation models, and / or navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225.

[0298] Server 1230 may include at least one processing unit 2020 configured to execute computer code or instructions stored in memory 2015 to perform various functions. For example, processing unit 2020 may analyze navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225, and generate an autonomous vehicle road navigation model based on that analysis. Processing unit 2020 may control communication unit 1405 to assign the autonomous vehicle road navigation model to one or more autonomous vehicles (e.g., one or more of vehicles 1205, 1210, 1215, 1220, and 1225, or any vehicle subsequently traveling on road segment 1200). Processing unit 2020 may be similar to or different from processors 180, 190, or processing unit 110.

[0299] Figure 21 A block diagram of memory 2015 is shown. Memory 2015 can store computer code or instructions for performing one or more operations to generate a road navigation model for autonomous vehicle navigation. Figure 21 As shown, memory 2015 may store one or more modules for performing operations for processing vehicle navigation information. For example, memory 2015 may include model generation module 2105 and model allocation module 2110. Processor 2020 may execute instructions stored in any of the modules 2105 and 2110 contained in memory 2015.

[0300] The model generation module 2105 may store instructions that, when executed by the processor 2020, may generate at least a portion of an autonomous vehicle road navigation model for a public road segment (e.g., road segment 1200) based on navigation information received from vehicles 1205, 1210, 1215, 1220, and 1225. For example, in generating the autonomous vehicle road navigation model, the processor 2020 may cluster vehicle trajectories along the public road segment 1200 into different clusters. The processor 2020 may determine a target trajectory along the public road segment 1200 based on the vehicle trajectories for each different cluster. Such an operation may involve finding the mean or average trajectory of the vehicle trajectories in each cluster (e.g., data representing the average of the vehicle trajectories in the cluster). In some embodiments, the target trajectory may be associated with a single lane of the public road segment 1200.

[0301] Road models and / or sparse maps can store trajectories associated with road segments. These trajectories, referred to as target trajectories, are provided to autonomous vehicles for autonomous navigation. Target trajectories can be received from multiple vehicles, or generated based on actual trajectories received from multiple vehicles, or recommended trajectories (actual trajectories with some modifications). The target trajectories contained in the road model or sparse map can be continuously updated (e.g., averaged) with new trajectories received from other vehicles.

[0302] Vehicles traveling on a road segment can collect data through various sensors. This data may include landmarks, road signature contours, vehicle motion (e.g., accelerometer data, speed data), vehicle position (e.g., GPS data), and can be used to reconstruct the actual trajectory itself, or the data may be transmitted to a server that reconstructs the vehicle's actual trajectory. In some embodiments, vehicles may transmit data to server 1230 related to the trajectory (e.g., curves in any reference frame), landmark data, and lane assignments along the driving path. Various vehicles traveling along the same road segment in multiple driving modes may have different trajectories. Server 1230 can identify routes or trajectories associated with each lane from the trajectories received from the vehicles through a clustering process.

[0303] Figure 22 The process of clustering vehicle trajectories associated with vehicles 1205, 1210, 1215, 1220, and 1225 to determine target trajectories for a public road segment (e.g., road segment 1200) is illustrated. The target trajectories, or multiple target trajectories, determined from the clustering process can be included in an autonomous vehicle road navigation model or a sparse map 800. In some embodiments, vehicles 1205, 1210, 1215, 1220, and 1225 traveling along road segment 1200 can transmit multiple trajectories 2200 to server 1230. In some embodiments, server 1230 can generate trajectories based on landmark, road geometry, and vehicle motion information received from vehicles 1205, 1210, 1215, 1220, and 1225. To generate an autonomous vehicle road navigation model, server 1230 can cluster vehicle trajectories 1600 into multiple clusters 2205, 2210, 2215, 2220, 2225, and 2230, as shown below. Figure 22 As shown.

[0304] Various criteria can be used to perform clustering. In some embodiments, all drivers in a cluster may be similar in terms of absolute heading along road segment 1200. Absolute heading can be obtained from GPS signals received by vehicles 1205, 1210, 1215, 1220, and 1225. In some embodiments, dead reckoning can be used to obtain absolute heading. As those skilled in the art will understand, dead reckoning can be used to determine the current position of vehicles 1205, 1210, 1215, 1220, and 1225 and thus their heading by using previously determined positions, estimated speeds, etc. Trajectories clustered by absolute heading can aid in route identification along the road.

[0305] In some embodiments, regarding lane assignments for driving along road segment 1200 (e.g., in the same lanes before and after an intersection), all driving in a cluster can be similar. The trajectory of lane assignment clustering can help identify lanes along the road. In some embodiments, both criteria (e.g., absolute heading and lane assignment) can be used for clustering.

[0306] Within each cluster 2205, 2210, 2215, 2220, 2225, and 2230, trajectories can be averaged to obtain a target trajectory associated with a specific cluster. For example, trajectories from multiple drives associated with the same lane cluster can be averaged. The averaged trajectory can be a target trajectory associated with a specific lane. To average a set of trajectories, server 1230 can select a reference frame for any trajectory C0. For all other trajectories (C1, ..., Cn), server 1230 can find a rigid transformation mapping Ci to C0, where i = 1, 2, ..., n, where n is a positive integer corresponding to the total number of trajectories included in the cluster. Server 1230 can compute the mean curve or trajectory in the C0 reference frame.

[0307] In some embodiments, landmarks can define arc length matching between different driving modes, which can be used for trajectory-lane alignment. In some embodiments, lane markings before and after intersections can be used for trajectory-lane alignment.

[0308] To assemble lanes from a trajectory, server 1230 can select a reference frame for any lane. Server 1230 can map partially overlapping lanes to the selected reference frame. Server 1230 can continue mapping until all lanes are in the same reference frame. Lanes adjacent to each other can be aligned as if they were the same lanes, and later they can be laterally shifted.

[0309] Landmarks identified along a road segment can be mapped to a common reference frame, first at the lane level and then at the intersection level. For example, the same landmark can be identified multiple times by multiple vehicles in multiple driving scenarios. The data received about the same landmark in different driving scenarios may differ slightly. Such data can be averaged and mapped to the same reference frame, such as the C0 reference frame. Additionally or alternatively, the variance of the data for the same landmark received in multiple driving scenarios can be calculated.

[0310] In some embodiments, each lane of road segment 120 may be associated with a target trajectory and certain landmarks. The target trajectory or multiple such target trajectories may be included in the autonomous vehicle road navigation model, which may later be used by other autonomous vehicles traveling along the same road segment 1200. Landmarks identified by vehicles 1205, 1210, 1215, 1220, and 1225 may be recorded in association with the target trajectory as the vehicles travel along road segment 1200. The target trajectory and landmark data may be continuously or periodically updated using new data received from other vehicles during subsequent driving.

[0311] For the localization of autonomous vehicles, the disclosed systems and methods may use an extended Kalman filter. Vehicle localization may be determined based on 3D position data and / or 3D orientation data, through the integral of self-motion, predicting the future location ahead of the vehicle's current location. Vehicle localization may be corrected or adjusted through image observation of landmarks. For example, when the vehicle detects a landmark within an image captured by a camera, the landmark may be compared with known landmarks stored in the road model or sparse map 800. Known landmarks may have known locations (e.g., GPS data) along the target trajectory stored in the road model and / or sparse map 800. Based on the landmark's current speed and image, the distance from the vehicle to the landmark can be estimated. The vehicle's localization along the target trajectory may be adjusted based on the distance to the landmark and the landmark's known location (stored in the road model or sparse map 800). The location / localization data of landmarks stored in the road model and / or sparse map 800 (e.g., the average from multiple drivers) may be assumed to be accurate.

[0312] In some embodiments, the disclosed system can form a closed-loop subsystem, wherein estimation of the vehicle's six degrees of freedom (DOF) positioning (e.g., three-dimensional position data plus three-dimensional orientation data) can be used for the autonomous vehicle's navigation (e.g., steering the autonomous vehicle's wheels) to reach a desired point (e.g., stored forward 1.3 seconds). In turn, data from steering and actual navigation measurements can be used to estimate the six degrees of freedom positioning.

[0313] In some embodiments, poles along the road, such as lampposts and power or cable poles, can be used as landmarks for locating vehicles. Other landmarks, such as traffic signs, traffic lights, arrows on the road, stop lines, and static features or signatures of objects along road segments, can also be used as landmarks for locating vehicles. When poles are used for location, the x-view of the pole (i.e., from the vehicle's perspective) can be used instead of the y-view (i.e., the distance to the pole), because the base of the pole may be obscured, and sometimes they are not on the road plane.

[0314] Figure 23 A navigation system for a vehicle is shown, which can be used for autonomous navigation using a crowdsourced sparse map. For illustration, the vehicle is referred to as vehicle 1205. Figure 23 The vehicle shown can be any other vehicle disclosed herein, including, for example, vehicles 1210, 1215, 1220, and 1225, as well as vehicle 200 shown in other embodiments. Figure 12 As shown, vehicle 1205 can communicate with server 1230. Vehicle 1205 may include image capture device 122 (e.g., camera 122). Vehicle 1205 may include navigation system 2300, which is configured to provide navigation guidance for vehicle 1205 traveling on a road (e.g., road segment 1200). Vehicle 1205 may also include other sensors, such as speed sensor 2320 and accelerometer 2325. Speed ​​sensor 2320 may be configured to detect the speed of vehicle 1205. Accelerometer 2325 may be configured to detect acceleration or deceleration of vehicle 1205. Figure 23 The vehicle 1205 shown can be an autonomous vehicle, and the navigation system 2300 can be used to provide navigation guidance for autonomous driving. Alternatively, the vehicle 1205 can also be a non-autonomous, human-controlled vehicle, and the navigation system 2300 can still be used to provide navigation guidance.

[0315] The navigation system 2300 may include a communication unit 2305 configured to communicate with a server 1230 via a communication path 1235. The navigation system 2300 may also include a GPS unit 2310 configured to receive and process GPS signals. The navigation system 2300 may also include at least one processor 2315 configured to process data such as GPS signals, map data from a sparse map 800 (which may be stored on storage provided on the vehicle 1205 and / or received from the server 1230), road geometry sensed by a road contour sensor 2330, images captured by a camera 122, and / or an autonomous vehicle road navigation model received from the server 1230. The road contour sensor 2330 may include different types of means for measuring different types of road contours (such as road surface roughness, road width, road height, road curvature, etc.). For example, the road contour sensor 2330 may include means for measuring the motion of the vehicle's suspension 2305 to derive a road roughness profile. In some embodiments, the road contour sensor 2330 may include a radar sensor to measure the distance from the vehicle 1205 to the roadside (e.g., an obstacle on the roadside), thereby measuring the width of the road. In some embodiments, the road contour sensor 2330 may include means configured to measure the vertical elevation of the road. In some embodiments, the road contour sensor 2330 may include means configured to measure the road curvature. For example, a camera (e.g., camera 122 or another camera) may be used to capture road images showing the road curvature. The vehicle 1205 can use such images to detect the road curvature.

[0316] At least one processor 2315 may be programmed to receive at least one environmental image associated with vehicle 1205 from camera 122. At least one processor 2315 may analyze the at least one environmental image to determine navigation information associated with vehicle 1205. The navigation information may include a trajectory associated with vehicle 1205 traveling along road segment 1200. At least one processor 2315 may determine the trajectory based on the motion of camera 122 (and thus the vehicle), such as three-dimensional translation and three-dimensional rotation. In some embodiments, at least one processor 2315 may determine the translation and rotation of camera 122 based on the analysis of multiple images acquired by camera 122. In some embodiments, the navigation information may include lane assignment information (e.g., which lane vehicle 1205 is traveling in along road segment 1200). The navigation information transmitted from vehicle 1205 to server 1230 may be used by server 1230 to generate and / or update an autonomous vehicle road navigation model, which may be transmitted back from server 1230 to vehicle 1205 for providing autonomous navigation guidance for vehicle 1205.

[0317] At least one processor 2315 may also be programmed to transmit navigation information from vehicle 1205 to server 1230. In some embodiments, the navigation information may be transmitted to server 1230 along with road information. The road positioning information may include at least one of GPS signals received by GPS unit 2310, landmark information, road geometry, lane information, etc. At least one processor 2315 may receive an autonomous vehicle road navigation model or a portion thereof from server 1230. The autonomous vehicle road navigation model received from server 1230 may include at least one update based on the navigation information transmitted from vehicle 1205 to server 1230. The model portion transmitted from server 1230 to vehicle 1205 may include an updated portion of the model. At least one processor 2315 may induce at least one navigation maneuver (e.g., steering, such as turning, braking, acceleration, passing another vehicle, etc.) of vehicle 1205 based on the received autonomous vehicle road navigation model or the updated portion of the model.

[0318] At least one processor 2315 may be configured to communicate with various sensors and components included in the vehicle 1205, including a communication unit 1705, a GPS unit 2315, a camera 122, a speed sensor 2320, an accelerometer 2325, and a road contour sensor 2330. At least one processor 2315 may collect information or data from the various sensors and components and transmit the information or data to the server 1230 via the communication unit 2305. Alternatively or additionally, the various sensors or components of the vehicle 1205 may also communicate with the server 1230 and transmit data or information collected by the sensors or components to the server 1230.

[0319] In some embodiments, vehicles 1205, 1210, 1215, 1220, and 1225 can communicate with each other and share navigation information, such that at least one of vehicles 1205, 1210, 1215, 1220, and 1225 can, for example, generate an autonomous vehicle road navigation model using crowdsourcing based on information shared by other vehicles. In some embodiments, vehicles 1205, 1210, 1215, 1220, and 1225 can share navigation information with each other, and each vehicle can update the autonomous vehicle road navigation model provided in its own vehicle. In some embodiments, at least one of vehicles 1205, 1210, 1215, 1220, and 1225 (e.g., vehicle 1205) can be used as a hub vehicle. At least one processor 2315 of the hub vehicle (e.g., vehicle 1205) can perform some or all of the functions performed by server 1230. For example, at least one processor 2315 of the hub vehicle can communicate with other vehicles and receive navigation information from other vehicles. At least one processor 2315 of the central vehicle can generate an autonomous vehicle road navigation model or an update to the model based on shared information received from other vehicles. The at least one processor 2315 of the central vehicle can transmit the autonomous vehicle road navigation model or the update to the model to other vehicles to provide autonomous navigation guidance.

[0320] Mapped lane signs and navigation based on mapped lane signs

[0321] As previously described, the autonomous vehicle road navigation model and / or sparse map 800 may include multiple mapped lane markings associated with road segments. These mapped lane markings can be used when the autonomous vehicle is navigating, as discussed in more detail below. For example, in some embodiments, the mapped lane markings can be used to determine the lateral position and / or orientation relative to a planned trajectory. Using this positional information, the autonomous vehicle can adjust its heading to match the orientation of the target trajectory at the determined location.

[0322] Vehicle 200 can be configured to detect lane markings in a given road segment. A road segment can include any markings on the road used to guide vehicular traffic. For example, lane markings can be solid or dashed lines distinguishing the edges of driving lanes. Lane markings can also include double lines, such as double solid lines, double dashed lines, or a combination of solid and dashed lines, to indicate, for example, whether passing is permitted in adjacent lanes. Lane markings can also include highway entrance and exit markings indicating, for example, deceleration lanes for exit ramps, or dashed lines indicating lanes for turning only or lanes nearing their end. Markings can also indicate work zones, temporary lane changes, routes through intersections, median strips, dedicated lanes (e.g., bicycle lanes, HOV lanes, etc.), or various other markings (e.g., pedestrian crossings, speed humps, railway crossgates, stop lines, etc.).

[0323] Vehicle 200 can use cameras, such as image capture devices 122 and 124 included in image acquisition unit 120, to capture images of surrounding lane markings. Vehicle 200 can analyze the images based on features identified in one or more captured images to detect point locations associated with lane markings. These point locations can be uploaded to a server to represent lane markings in sparse map 800. Depending on the camera position and field of view, lane markings on both sides of the vehicle can be detected simultaneously from a single image. In other embodiments, different cameras can be used to capture images on multiple sides of the vehicle. Lane markings can be stored as splines or a series of points in sparse map 800 instead of uploading actual images of these markings, thereby reducing the size of sparse map 800 and / or the data that must be remotely uploaded by the vehicle.

[0324] Figures 24A-24D An exemplary point location that can be detected by vehicle 200 to represent a specific lane sign is shown. Similar to landmarks as described above, vehicle 200 can use various image recognition algorithms or software to identify point locations within a captured image. For example, vehicle 200 can identify a series of edge points, corner points, or various other point locations associated with a specific lane sign. Figure 24A A solid lane marking 2410 is shown that can be detected by vehicle 200. Lane marking 2410 can indicate the outer edge of the road, represented by a solid white line. Figure 24AAs shown, vehicle 200 can be configured to detect multiple edge localization points 2411 along lane markings. Localization points 2411 can be collected to represent lane markings at any interval sufficient to create a mapped lane marking in a sparse map. For example, lane markings can be represented by one point per meter of detected edges, one point every five meters of detected edges, or at other suitable intervals. In some embodiments, the interval can be determined by other factors, rather than by a set interval, such as, for example, based on the points where vehicle 200 has the highest confidence in the localization of the detected points. Although Figure 24A The diagram shows edge positioning points on the inner edge of lane sign 2410, which can be collected on the outer edge of the line or along both edges. Furthermore, although in Figure 24A The diagram shows a single line, but similar edge points can be detected for double solid lines. For example, point 2411 can be detected along the edge of one or two solid lines.

[0325] Depending on the type or shape of the lane markings, vehicle 200 may also represent lane markings differently. Figure 24B An exemplary dashed lane marking 2420 is shown that can be detected by vehicle 200. The vehicle can detect a series of corner points 2421 representing the corners of the dashed lane markings to define the complete boundary of the dashed line, rather than as... Figure 24A How edge points are identified in the same way. Although Figure 24B Each corner of a given dashed line marker is shown, but vehicle 200 can detect or upload a subset of the points shown in the diagram. For example, vehicle 200 can detect the leading edge or leading corner of a given dashed line marker, or it can detect the two corner points closest to the inside of the lane. Furthermore, not every dashed line marker can be captured; for example, vehicle 200 can capture and / or record points representing samples of dashed line markers (e.g., every one, every three, every five, etc.), or points representing dashed line markers at predetermined intervals (e.g., every meter, every five meters, every ten meters, etc.). Corner points of similar lane markings (such as signs indicating that the lane is for an exit ramp, signs indicating the end of a specific lane, or various other lane markings that may have detectable corner points) can also be detected. Corner points of lane markings consisting of double dashed lines or a combination of solid and dashed lines can also be detected.

[0326] In some embodiments, the points uploaded to the server to generate mapped lane signs can represent points other than detected edge points or corner points. Figure 24CA series of points that can represent the centerline of a given lane marking are shown. For example, a solid lane 2410 can be represented by centerline point 2441 along the centerline 2440 of the lane marking. In some embodiments, vehicle 200 can be configured to detect these center points using various image recognition techniques, such as convolutional neural networks (CNNs), scale-invariant feature transforms (SIFTs), histogram of oriented gradients (HOG) features, or other techniques. Alternatively, vehicle 200 can detect other points, such as Figure 24A The edge point 2411 is shown, and the centerline point 2441 can be calculated, for example, by detecting points along each edge and determining the midpoint between the edge points. Similarly, the dashed lane marking 2420 can be represented by the centerline point 2451 along the centerline 2450 of the lane marking. The centerline point can be located at the edge of the dashed line, as shown in Figure 24C, or at various other locations along the centerline. For example, each dashed line can be represented by a single point at the geometric center of the dashed line. These points can also be spaced apart along the centerline at predetermined intervals (e.g., every meter, every 5 meters, every 10 meters, etc.). The centerline point 2451 can be directly detected by the vehicle 200, or it can be calculated based on other detected reference points (such as corner point 2421), such as... Figure 24B As shown. Using a technique similar to that described above, the center line can also be used to indicate other lane marking types, such as double lines.

[0327] In some embodiments, vehicle 200 may identify points representing other features, such as vertices between two intersecting lane signs. Figure 24D An exemplary point representing an intersection between two lane signs 2460 and 2465 is shown. Vehicle 200 can calculate vertex 2466 representing the intersection between the two lane signs. For example, one of lane signs 2460 or 2465 may represent a train crossing area or other crossing area in a road segment. Although lane signs 2460 and 2465 are shown to intersect each other perpendicularly, various other configurations can be detected. For example, lane signs 2460 and 2465 may intersect at other angles, or one or both lane signs may terminate at vertex 2466. Similar techniques can also be applied to intersections between dashed lines or other lane sign types. In addition to vertex 2466, various other points 2467 can be detected, providing further information about the orientation of lane signs 2460 and 2465.

[0328] Vehicle 200 can associate real-world coordinates with each detected point of the lane sign. For example, a location identifier (including the coordinates of each point) can be generated and uploaded to a server for mapping lane signs. The location identifier can also include other identifying information about the point, including whether the point represents a corner, edge, center, etc. Therefore, vehicle 200 can be configured to determine the real-world location of each point based on analysis of an image. For example, vehicle 200 can detect other features in the image (such as various landmarks as described above) to locate the real-world location of the lane sign. This can include determining the location of the lane sign in the image relative to the detected landmarks, or determining the vehicle's position based on the detected landmarks and then determining the distance from the vehicle (or the vehicle's target trajectory) to the lane sign. When landmarks are unavailable, the location of the lane sign point can be determined relative to the vehicle's position determined based on dead reckoning. The real-world coordinates included in the location identifier can be represented as absolute coordinates (e.g., latitude / longitude coordinates) or relative to other features, such as based on longitudinal position along the target trajectory and lateral distance from the target trajectory. The location identifiers can then be uploaded to a server for use in generating mapped lane markings in a navigation model (such as a sparse map 800). In some embodiments, the server can construct splines representing lane markings for road segments. Alternatively, the vehicle 200 can generate splines and upload them to the server for recording in the navigation model.

[0329] Figure 24E An exemplary navigation model or sparse map is shown, including corresponding road segments with mapped lane markings. The sparse map may include a target trajectory 2475 followed by a vehicle along the road segment. As described above, the target trajectory 2475 may represent the ideal path taken by the vehicle when traveling on the corresponding road segment, or it may be located elsewhere on the road (e.g., the centerline of the road, etc.). The target trajectory 2475 can be calculated using various methods as described above, for example, based on an aggregation (e.g., a weighted combination) of two or more reconstructed trajectories of the vehicle traversing the same road segment.

[0330] In some embodiments, target trajectories can be generated equally for all vehicle types and for all road, vehicle, and / or environmental conditions. However, in other embodiments, various other factors or variables may also be considered when generating target trajectories. Different target trajectories may be generated for different types of vehicles (e.g., cars, light trucks, and trailers). For example, a target trajectory with a relatively small turning radius may be generated for a small car compared to a larger semi-trailer truck. In some embodiments, road, vehicle, and environmental conditions may also be considered. For example, different target trajectories may be generated for different road conditions (e.g., wet, snowy, icy, dry, etc.), vehicle conditions (e.g., tire condition or estimated tire condition, braking condition or estimated braking condition, remaining fuel, etc.), or environmental factors (e.g., time of day, visibility, weather, etc.). Target trajectories may also depend on one or more aspects or characteristics of a particular road segment (e.g., speed limit, turning frequency and size, gradient, etc.). In some embodiments, various user settings may also be used to determine the target trajectory, such as set driving modes (e.g., desired driving aggressiveness, economy mode, etc.).

[0331] The sparse map may also include mapped lane markers 2470 and 2480 representing lane markings along road segments. The mapped lane markers may be represented by multiple location identifiers 2471 and 2481. As described above, the location identifiers may include the location of the point associated with the detected lane marker in real-world coordinates. Similar to a target trajectory in the model, lane markers may also include elevation data and may be represented as curves in three-dimensional space. For example, the curve may be a spline connecting three-dimensional polynomials of appropriate order. The curve may be computed based on the location identifiers. The mapped lane markers may also include other information or metadata about the lane markers, such as identifiers of the type of lane marker (e.g., lane marker between two lanes traveling in the same direction, lane marker between two lanes traveling in opposite directions, curb, etc.) and / or other characteristics of the lane markers (e.g., solid line, dashed line, single line, double line, yellow, white, etc.). In some embodiments, for example, the mapped lane markers may be continuously updated within the model using crowdsourcing techniques. The same vehicle can upload location identifiers at multiple times while traveling on the same road segment, or it can select data from multiple vehicles (such as 1205, 1210, 1215, 1220, and 1225) traveling on the same road segment at different times. The sparse map 800 can then be updated or refined based on subsequent location identifiers received from the vehicles and stored in the system. As the mapped lane markings are updated and refined, the updated road navigation model and / or sparse map can be assigned to multiple autonomous vehicles.

[0332] Generating mapped lane markings in sparse maps can also include detecting and / or mitigating errors based on anomalies in the image or in the actual lane markings themselves. Figure 24F An exemplary anomaly 2495 associated with the detection of lane sign 2490 is shown. Anomaly 2495 may appear in an image captured by vehicle 200 from, for example, objects obstructing the camera's view of the lane sign, debris on the lens, etc. In some cases, the anomaly may be due to the lane sign itself; the lane sign may be damaged or worn, or partially covered by dust, debris, water, snow, or other materials on the road. Anomaly 2495 may result in an error point 2491 detected by vehicle 200. Sparse map 800 can provide correct mapping of lane signs and eliminate errors. In some embodiments, vehicle 200 may detect error point 2491, for example, by detecting anomaly 2495 in the image, or by identifying errors based on detected lane sign points before and after the anomaly. Based on the detection of the anomaly, vehicle may omit point 2491, or may adjust it to be consistent with other detected points. In other embodiments, for example, the error can be corrected after the point has been uploaded by determining that the point is outside an expected threshold based on other points uploaded during the same trip or based on aggregation of data from previous trips along the same road segment.

[0333] Mapped lane markers in the navigation model and / or sparse map can also be used for navigation of autonomous vehicles traversing corresponding roads. For example, a vehicle navigating along a target trajectory can periodically use mapped lane markers in the sparse map to align itself with the target trajectory. As mentioned above, between landmarks, vehicles can navigate based on dead reckoning, where the vehicle uses sensors to determine its own motion and estimate its position relative to the target trajectory. Errors can accumulate over time, and the vehicle's position determination relative to the target trajectory may become increasingly inaccurate. Accordingly, vehicles can use lane markers appearing in the sparse map 800 (and their known locations) to reduce errors in position determination caused by dead reckoning. In this way, the identified lane markers included in the sparse map 800 can serve as navigation anchors from which the vehicle's accurate position relative to the target trajectory can be determined.

[0334] Figure 25A An exemplary image 2500 of the vehicle's surrounding environment, which can be used for navigation based on mapped lane signs, is shown. Image 2500 can be captured, for example, by vehicle 200 via image capture devices 122 and 124 included in image acquisition unit 120. Image 2500 may include an image of at least one lane sign 2510, such as... Figure 25A As shown. Image 2500 may also include one or more landmarks 2521, such as road signs, for navigation as described above. Figure 25ASome elements (such as elements 2511, 2530 and 2520) that are not present in the captured image 2500 but are detected and / or identified by the vehicle 200 are also shown for reference.

[0335] Use the above reference Figures 24A-24D and Figure 24F The various techniques described allow a vehicle to analyze image 2500 to identify lane markings 2510. Various points 2511 corresponding to features of the lane markings in the image can be detected. For example, point 2511 may correspond to the edge of a lane marking, a corner of a lane marking, the midpoint of a lane marking, a vertex between two intersecting lane markings, or various other features or locations. Point 2511 can be detected as corresponding to locations of points stored in a navigation model received from a server. For example, if a sparse map containing points representing the centerline of mapped lane markings is received, point 2511 can also be detected based on the centerline of lane marking 2510.

[0336] The vehicle can also be configured to determine its longitudinal position, represented by element 2520, and located along a target trajectory. For example, the longitudinal position 2520 can be determined from image 2500 by detecting landmarks 2521 within image 2500 and comparing the measured position with known landmark positions stored in the road model or sparse map 800. The vehicle's position along the target trajectory can then be determined based on the distance to the landmark and the known position of the landmark. The longitudinal position 2520 can also be determined from images other than those used to determine the position of lane markings. For example, the longitudinal position 2520 can be determined by detecting landmarks in images taken simultaneously or nearly simultaneously with image 2500 from other cameras within image acquisition unit 120. In some cases, the vehicle may not be near any landmarks or other reference points used to determine the longitudinal position 2520. In this case, the vehicle can navigate based on dead reckoning and thus use sensors to determine its own motion and estimate its longitudinal position 2520 relative to the target trajectory. The vehicle can also be configured to determine a distance 2530, representing the actual distance between the vehicle and lane sign 2510 observed in the captured images(s). When determining the distance 2530, camera angle, vehicle speed, vehicle width, or various other factors may be taken into account.

[0337] Figure 25BLateral positioning correction for a vehicle based on lane markings mapped in a road navigation model is illustrated. As described above, vehicle 200 can use one or more images captured by vehicle 200 to determine the distance 2530 between vehicle 200 and lane markings 2510. Vehicle 200 can also access a road navigation model (such as a sparse map 800), which may include mapped lane markings 2550 and a target trajectory 2555. The mapped lane markings 2550 can be modeled using techniques described above, such as using crowdsourced location identifiers captured by multiple vehicles. The target trajectory 2555 can also be generated using various techniques previously described. Vehicle 200 can also be configured to determine or estimate the longitudinal position 2520 along the target trajectory 2555, as referenced above. Figure 25A The vehicle 200 can then determine the expected distance 2540 based on the lateral distance between the target trajectory 2555 and the mapped lane markings 2550 corresponding to the longitudinal position 2520. The lateral positioning of the vehicle 200 can be corrected or adjusted by comparing the actual distance 2530 measured using the captured images(s) with the expected distance 2540 from the model.

[0338] Figure 26A This is a flowchart illustrating an exemplary process 2600A for mapping lane markings for autonomous vehicle navigation, consistent with the disclosed embodiments. In step 2610, process 2600A may include receiving two or more location identifiers associated with detected lane markings. For example, step 2610 may be performed by server 1230 or one or more processors associated with that server. The location identifiers may include the location of a point associated with the detected lane marking in real-world coordinates, as referenced above. Figure 24EAs described above. In some embodiments, the location identifier may also include other data, such as additional information about road segments or lane markings. Additional data may also be received during step 2610, such as accelerometer data, speed data, landmark data, road geometry or contour data, vehicle positioning data, self-motion data, or various other forms of data as described above. The location identifier may be generated by a vehicle (such as vehicles 1205, 1210, 1215, 1220, and 1225) based on images captured by the vehicle. For example, the identifier may be determined based on acquiring at least one image representing the environment of the main vehicle from a camera associated with the main vehicle, analyzing the at least one image to detect lane markings in the main vehicle's environment, and analyzing the at least one image to determine the position of the detected lane markings relative to the positioning associated with the main vehicle. As described above, lane markings may include various different marking types, and the location identifier may correspond to various points relative to the lane markings. For example, in the case where the detected lane marking is part of a dashed line marking the lane boundary, these points may correspond to the detected corner of the lane marking. In the case where the detected lane marking is part of a solid line marking the lane boundary, these points may correspond to the detected edge of the lane marking with various spacings as described above. In some embodiments, these points may correspond to the center line of a detected lane marking, such as Figure 24C As shown, or it may correspond to at least one of the vertices between two intersecting lane signs and two other points associated with the intersecting lane signs, such as Figure 24D As shown.

[0339] In step 2612, process 2600A may include associating the detected lane sign with a corresponding road segment. For example, server 1230 may analyze real-world coordinates or other information received during step 2610 and compare the coordinates or other information with positioning information stored in the autonomous vehicle road navigation model. Server 1230 may determine the road segment in the model corresponding to the real-world road segment where the lane sign was detected.

[0340] In step 2614, process 2600A may include updating the autonomous vehicle road navigation model relative to the corresponding road segment based on two or more location identifiers associated with the detected lane markings. For example, the autonomous road navigation model may be a sparse map 800, and server 1230 may update the sparse map to include or adjust the mapped lane markings in the model. Server 1230 may base its updates on the above references. Figure 24EVarious methods or processes are described to update the model. In some embodiments, updating the autonomous vehicle road navigation model may include storing one or more location indicators of detected lane markings in real-world coordinates. The autonomous vehicle road navigation model may also include at least one target trajectory followed by the vehicle along a corresponding road segment, such as... Figure 24E As shown.

[0341] In step 2616, process 2600A may include assigning the updated autonomous vehicle road navigation model to multiple autonomous vehicles. For example, server 1230 may assign the updated autonomous vehicle road navigation model to vehicles 1205, 1210, 1215, 1220, and 1225, which can use the model for navigation. The autonomous vehicle road navigation model may be assigned via one or more networks (e.g., via cellular networks and / or the Internet, etc.) through wireless communication path 1235, such as... Figure 12 As shown.

[0342] In some embodiments, lane markings can be mapped using data received from multiple vehicles (such as through crowdsourcing technologies), as referenced above. Figure 24E The process 2600A may include, for example, receiving a first communication from a first master vehicle including a location identifier associated with the detected lane sign, and a second communication from a second master vehicle including an additional location identifier associated with the detected lane sign. For example, the second communication may be received from a subsequent vehicle traveling on the same road segment, or from the same vehicle on a subsequent journey along the same road segment. The process 2600A may also include refining the determination of at least one location associated with the detected lane sign based on the location identifier received in the first communication and based on the additional location identifier received in the second communication. This may include using the averaging and / or filtering out “ghost” identifiers that may not reflect the real-world location of the lane sign using multiple location identifiers.

[0343] Figure 26B This is a flowchart illustrating an exemplary process 2600B for autonomously navigating a host vehicle along a road segment using mapped lane markings. Process 2600B can be executed, for example, by the processing unit 110 of the autonomous vehicle 200. In step 2620, process 2600B may include receiving an autonomous vehicle road navigation model from a server-based system. In some embodiments, the autonomous vehicle road navigation model may include a target trajectory of the host vehicle along a road segment and location identifiers associated with one or more lane markings associated with the road segment. For example, vehicle 200 may receive a sparse map 800 or another road navigation model generated using process 2600A. In some embodiments, the target trajectory may be represented as a three-dimensional spline, for example, as... Figure 9B As shown in the reference above. Figures 24A-24F The location identifier may include the location of the point associated with the lane sign in real-world coordinates (e.g., the corner point of the dashed lane sign, the edge point of the solid lane sign, the vertex between two intersecting lane signs, and other points associated with intersecting lane signs, the center line associated with the lane sign, etc.).

[0344] In step 2621, process 2600B may include receiving at least one image representing the environment of the vehicle. This image may be received from an image capturing device of the vehicle (such as image capturing devices 122 and 124 included in image acquisition unit 120). The image may include images of one or more lane markings, similar to image 2500 as described above.

[0345] In step 2622, process 2600B may include determining the longitudinal position of the master vehicle along the target trajectory. (Refer to the above.) Figure 25A This can be based on other information in the captured images (e.g., landmarks, etc.) or by dead reckoning between detected landmarks.

[0346] In step 2623, process 2600B may include determining the expected lateral distance to the lane sign based on the determined longitudinal position of the master vehicle along the target trajectory and based on two or more location identifiers associated with at least one lane sign. For example, vehicle 200 may use sparse map 800 to determine the expected lateral distance to the lane sign. Figure 25B As shown, the longitudinal position 2520 along the target trajectory 2555 can be determined in step 2622. Using the sparse map 800, the vehicle 200 can determine the expected distance 2540 to the mapped lane marker 2550 corresponding to the longitudinal position 2520.

[0347] In step 2624, process 2600B may include analyzing at least one image to identify at least one lane sign. For example, vehicle 200 may use various image recognition techniques or algorithms to identify lane signs within the image, as described above. For example, lane sign 2510 may be detected through image analysis of image 2500, such as... Figure 25A As shown.

[0348] In step 2625, process 2600B may include determining the actual lateral distance to at least one lane sign based on analysis of at least one image. For example, the vehicle may determine a distance 2530 representing the actual distance between the vehicle and lane sign 2510, such as... Figure 25A As shown. When determining the distance 2530, factors such as camera angle, vehicle speed, vehicle width, camera position relative to the vehicle, and various other factors can be considered.

[0349] In step 2626, process 2600B may include determining the autonomous steering action of the master vehicle based on the difference between the expected lateral distance to at least one lane sign and the determined actual lateral distance to at least one lane sign. For example, as referenced above... Figure 25B The vehicle 200 can compare the actual distance 2530 with the expected distance 2540. The difference between the actual distance and the expected distance can indicate the error (and its magnitude) between the vehicle's actual position and the target trajectory the vehicle is to follow. Accordingly, the vehicle can determine autonomous steering actions or other autonomous actions based on this difference. For example, if the actual distance 2530 is less than the expected distance 2540, such as... Figure 25B As shown, the vehicle can determine an autonomous steering action to guide it away from lane marker 2510. Therefore, the vehicle's position relative to the target trajectory can be corrected. Process 2600B can be used, for example, to improve vehicle navigation between landmarks.

[0350] Map management using Horizon (electronic horizon)

[0351] Despite increased processing power and storage capacity and reduced costs, there is still a desire to use them more efficiently. The systems and methods disclosed herein allow vehicles to dynamically receive and load map data relevant to their travel routes, rather than loading large map datasets that the vehicle may not use during the trip. In doing so, the systems and methods reduce the vehicle's hardware requirements by receiving and processing only the map data the vehicle might need. Furthermore, the systems and methods can also allow for reduced transmission costs of data exchanged between the vehicle and, for example, a central server deploying the map data. Additionally, the disclosed systems and methods can allow the vehicle to receive the most up-to-date map data that the vehicle may need more frequently. For example, the systems and methods can determine the vehicle's potential travel area (or potential travel envelope) based on navigation information such as the vehicle's position, speed, and direction of travel. The systems and methods can also be configured to determine one or more road segments associated with the potential travel area from the vehicle and transmit map data associated with the road segments to the vehicle. The vehicle (and / or the driver) can then navigate based on the received map data.

[0352] Figure 27 An exemplary system 2700 for providing one or more map segments to one or more vehicles, consistent with the disclosed embodiments, is shown. Figure 27As shown, system 2700 may include server 2701, one or more vehicles 2702, one or more vehicle devices 2703 associated with the vehicles, database 2704, and network 2705. Server 2701 may be configured to provide one or more map segments to one or more vehicles (and / or one or more vehicle devices associated with the vehicles) based on navigation information received from one or more vehicles. For example, vehicles 2702 and / or vehicle devices 2703 may be configured to collect navigation information and transmit it to server 2701. Server 2701 may transmit one or more map segments to vehicles 2702 and / or vehicle devices 2703, the one or more map segments including map information for a geographic area based on the received navigation information. Database 2704 may be configured to store information for storing components of system 2700 (e.g., server 2701, vehicles 2702, and / or vehicle devices 2703). Network 2705 may be configured to facilitate communication between components of system 2700.

[0353] Server 2701 may be configured to receive navigation information from vehicle 2702 (and / or vehicle equipment 2703). In some embodiments, the navigation information may include the position of vehicle 2702, the speed of vehicle 2702, and the direction of travel of vehicle 2702. Server 2701 may also be configured to analyze the received navigation information and determine the potential driving envelope of vehicle 2702. The potential driving envelope of vehicle may be a region surrounding vehicle. For example, the potential driving envelope of vehicle may include a region covering: a first predetermined distance from vehicle in the driving direction of vehicle, a second predetermined distance from vehicle in the opposite direction of vehicle driving, a third predetermined distance from vehicle to the left of vehicle, and a fourth predetermined distance from vehicle to the right of vehicle. In some embodiments, the first predetermined distance from vehicle in the driving direction of vehicle may include a predetermined distance in front of vehicle, which may constitute the electronic horizon of vehicle. In some embodiments, the potential driving envelope of vehicle may include one or more distances (one, two, three, ..., n) from vehicle relative to its current position in one or more (or all) possible driving directions of vehicle. For example, on a road where a vehicle can make a U-turn, the vehicle's potential driving envelope may include, in addition to a predetermined distance in the forward direction, a predetermined distance in the opposite direction from the vehicle, since the vehicle may perform a U-turn and can (typically) navigate in the direction opposite to its current direction of motion. As another example, if it is impossible to travel in the opposite direction from the current location (e.g., due to physical obstacles) and a U-turn is impossible within a certain distance ahead of the current location, the potential driving envelope may not include the distance in the opposite direction. Similar to the actual horizon in the real world, the electronic horizon may be correlated with the vehicle's potential driving distance within a certain time window based on the main vehicle's current speed and current direction of travel. Server 2701 may also be configured to send one or more map segments to the vehicle, which include map information of geographic areas that at least partially overlap with the potential driving envelope of vehicle 2702.

[0354] In some embodiments, server 2701 may be a cloud server performing the functions disclosed herein. The term "cloud server" refers to a computer platform that provides services via a network (e.g., the Internet). In this example configuration, server 2701 may use virtual machines that do not correspond to separate hardware. For example, computing and / or storage capacity may be implemented by allocating appropriate portions of the required computing / storage capacity from a scalable repository such as a data center or distributed computing environment. In one example, server 2701 may implement the methods described herein using custom hardwired logic, one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), firmware, and / or program logic, which, together with the computer system, make server 2701 a dedicated machine.

[0355] Vehicle 2702 and / or vehicle equipment 2703 may be configured to collect navigation information and transmit it to server 2701. For example, vehicle 2702 and / or vehicle equipment 2703 may be configured to receive data from one or more sensors and determine navigation information such as the vehicle's position, speed, and / or driving direction based on the received data. In some embodiments, the navigation information may include sensor data received from one or more sensors associated with vehicle 3302 (e.g., from GPS devices, speed sensors, accelerometers, suspension sensors, cameras, LiDAR devices, Vision Detection and Ranging (VIDAR) devices, etc., or combinations thereof). Vehicle 2702 and / or vehicle equipment 2703 may also be configured to transmit navigation information to server 2701 via, for example, network 2705. Alternatively or additionally, vehicle 2702 and / or vehicle equipment 2703 may be configured to transmit sensor data to server 2701. Vehicle 2702 and / or vehicle equipment 2703 may also be configured to receive map information from server 2701 via, for example, network 2705. The map information may include data relating to the location of various items in a reference coordinate system, including, for example, roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, sparse data models including polynomial representations of certain road features (e.g., lane markings), the target trajectory of the main vehicle, etc., or combinations thereof. In some embodiments, vehicle 2702 and / or vehicle equipment 2703 may be configured to plan route paths and / or navigate vehicle 2702 based on the map information. For example, vehicle 2702 and / or vehicle equipment 2703 may be configured to determine a route to a destination based on the map information. Alternatively or additionally, vehicle 2702 and / or vehicle equipment 2703 may be configured to perform at least one navigation action (e.g., turning, stopping at a location, etc.) based on the received map information. In some embodiments, vehicle 2702 may include equipment with a similar configuration and / or performing similar functions to system 100 described above. Alternatively or additionally, vehicle equipment 2703 may have a similar configuration to the system 100 described above and / or perform similar functions.

[0356] Database 2704 may include a map database configured to store map data for components of system 2700 (e.g., server 2701, vehicle 2702, and / or vehicle equipment 2703). In some embodiments, server 2701, vehicle 2702, and / or vehicle equipment 2703 may be configured to access database 2704 via network 2705 and retrieve stored data from and / or upload data to database 2704. For example, server 2701 may transfer data associated with one or more road navigation models to database 2704 for storage. Vehicle 2702 and / or vehicle equipment 2703 may download road navigation models from database 2704. In some embodiments, database 2704 may include data relating to the location of various items in a reference coordinate system, including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, etc., or combinations thereof. In some embodiments, database 2704 may include a database similar to map database 160 described elsewhere in this disclosure.

[0357] Network 2705 can be any type of network (including infrastructure) that provides communication, exchanges information, and / or facilitates information exchange between components of system 2700. For example, network 2705 may include or be part of networks such as the Internet, a local area network (LAN), a wireless network (e.g., Wi-Fi / 302.11), or other suitable connections. In other embodiments, one or more components of system 2700 may communicate directly via a dedicated communication link, such as a telephone network, an extranet, an intranet, the Internet, satellite communications, offline communications, wireless communications, repeater communications, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), etc.

[0358] As described elsewhere in this disclosure, vehicle 2702 may transmit navigation information to server 2701 via network 2705. Server 2701 may analyze the navigation information received from vehicle 2702 and determine a potential driving envelope of vehicle 2702 based on the analysis of the navigation information. The potential driving envelope may cover the location of vehicle 2702. In some embodiments, the potential driving envelope may include a boundary. The boundary of the potential driving envelope may have a shape, including, for example, a triangular shape, a quadrilateral shape, a parallelogram shape, a rectangular shape, a square (or approximately square) shape, a trapezoidal shape, a rhombus shape, a hexagonal shape, an octagonal shape, a circular (or approximately circular) shape, an elliptical shape, an egg-shaped shape, an irregular shape, etc., or combinations thereof. Figures 28A-28D An exemplary potential driving envelope of a vehicle in region 2800, consistent with the disclosed embodiments, is shown. Figure 28AAs shown, server 2701 can determine a potential driving envelope with boundary 2811 for vehicle 2702, which may include a trapezoidal shape. As another example, such as... Figure 28B As shown, server 2701 can determine a potential driving envelope for vehicle 2702 with boundary 2812, which may include an elliptical shape. As another example, as shown in FIG28, server 2701 can determine a potential driving envelope for vehicle 2702 with boundary 2813, which may include a triangular shape. As another example, as shown in FIG28, server 2701 can determine a potential driving envelope for vehicle 2702 with boundary 2814, which may include a rectangular shape. Alternatively or additionally, the shape of the potential driving envelope may have boundaries determined by one or more potential paths on which the vehicle can travel from its position (e.g., its current position). Those skilled in the art will understand that the shape of the potential driving envelope is not limited to the exemplary shapes described in this disclosure. Other shapes are also possible. For example, the potential driving envelope may include irregular shapes (e.g., determined based on one or more boundaries of a jurisdiction such as a country, state, county, city, and / or road) and / or portions of any shape described herein.

[0359] As described elsewhere in this disclosure, server 2701 may also be configured to transmit one or more map segments to vehicle 2702, the one or more map segments including map information of geographic areas that at least partially overlap with the potential driving envelope of the vehicle. In some embodiments, the one or more map segments transmitted to vehicle 2702 may include one or more tiles representing areas with predetermined areas. The size and / or shape of the tiles may vary. In some embodiments, the area of ​​a tile in an area may range from 0.25 to 100 square kilometers, which may be limited to sub-ranges of 0.25 square kilometers to 1 square kilometer, 1 square kilometer to 10 square kilometers, 10 to 25 square kilometers, 25 to 50 square kilometers, and 50 to 100 square kilometers. In some embodiments, the predetermined area of ​​a tile may be less than or equal to ten square kilometers. Alternatively, the predetermined area of ​​a tile may be less than or equal to one square kilometer. Alternatively, the predetermined area of ​​a tile may be less than or equal to ten square kilometers. Alternatively or additionally, the size of the tile may vary based on the type of area in which the tile is located. For example, the tile size in rural areas or areas with fewer roads can be larger than that in urban areas or areas with more roads. In some embodiments, the tile size can be determined based on the information density around the vehicle's current location (or the location where the data is obtained), the vehicle's possible paths, the number of possible routes in the area, the type of routes in the area and general navigation modes (e.g., primary, urban, rural, dirt, etc.), and / or trends in the relevant area. For example, if a large percentage of vehicles typically stay on the primary roads in a particular area or region, map information along a short distance along a secondary road can be obtained. If the vehicle actually navigates to a secondary road that is typically less traveled, more map information for the secondary road (e.g., map information along a longer distance along a side road) can be obtained and / or transmitted to the vehicle. In some embodiments, tiles can have rectangular, square, hexagonal, etc., or combinations thereof. Those skilled in the art will understand that the shape of the tiles is not limited to the shapes described in this disclosure. For example, tiles can include irregular shapes (e.g., determined by at least one boundary of a jurisdiction (state, county, city, or town) or other area (e.g., street, highway). Alternatively or additionally, a block may include portions of any shape disclosed herein.

[0360] Figures 28E-28H Exemplary map tiles representing the potential driving envelope of a vehicle consistent with the disclosed embodiments are shown. Figures 28E-28H As shown, server 2701 can divide region 2800 (or a smaller or larger region) into multiple tiles 2811. Server 2701 can also be configured to identify one or more tiles that at least partially overlap with the potential driving envelope of vehicle 2702. For example, as Figure 28EAs shown, server 2701 can determine region 2831, which has tiles that intersect with or are within the boundary 2821 of the potential driving envelope of vehicle 2702. As another example, such as... Figure 28F As shown, server 2701 can determine region 2832, which has tiles that intersect with or are within the boundary 2822 of the potential driving envelope of vehicle 2702. As another example, such as... Figure 28G As shown, server 2701 can determine region 2833, which has tiles that intersect with or are within the boundary 2823 of the potential driving envelope of vehicle 2702. As another example, such as... Figure 28H As shown, server 2701 can determine region 2834, which has tiles intersecting with or within the boundary 2824 of the potential driving envelope of vehicle 2702. In some embodiments, server 2701 can transmit map information and / or data relating to one or more road segments in the determined region to vehicle 2702.

[0361] Figure 29A and Figure 29B Exemplary map tiles consistent with the disclosed embodiments are shown. For example... Figure 29A As shown, a region (or map) can be divided into multiple tiles at different levels. For example, in some embodiments, a region can be divided into multiple tiles at level 1, and each tile at level 1 can be divided into multiple tiles at level 2. Each tile at level 2 can be divided into multiple tiles at level 3, and so on. Figure 29B This illustrates multiple tiles within a region. Alternatively or additionally, a region or country may be divided into multiple tiles based on jurisdiction (e.g., state, county, city, town) and / or other areas (e.g., street, highway). In some embodiments, the area of ​​the tiles can vary. For example, as... Figure 29A and 29B As shown, a region (or map) can be divided into different levels, and tiles at a specific level can have a specific area.

[0362] In some embodiments, tiles can be presented in a data blob, which may include metadata blocks (e.g., 64 bytes), signature blocks (e.g., 256 bytes), and encoded map data blocks (e.g., various sizes in MapBox format).

[0363] In some embodiments, server 2701 may retrieve data associated with one or more tiles in the area and transmit the data to vehicle 2702 via, for example, network 2705.

[0364] Alternatively or additionally, vehicle 2702 may retrieve data associated with one or more road segments from a storage device. For example, vehicle 2702 may receive one or more road segments from server 2701, as described elsewhere in this disclosure. Vehicle 2702 may also store the received one or more road segments in local storage and load one or more road segments contained in the one or more road segments into memory for processing. Alternatively, vehicle 2702 may include local storage configured to store one or more road segments and retrieve data associated with the one or more road segments from the local storage, instead of receiving one or more road segments from server 2701.

[0365] In some embodiments, vehicle 2702 can acquire data (e.g., map information) related to one or more tiles based on its location. For example, vehicle 2702 can determine its current location (as described elsewhere in this disclosure) and identify the first tile where the current location is located. Vehicle 2702 can also acquire the first tile and one or more (or all) tiles adjacent to the first tile. Alternatively or additionally, vehicle 2702 can acquire one or more (or all) tiles within a predetermined distance from the first tile. Alternatively or additionally, vehicle 2702 can acquire one or more (or all) tiles within a predetermined separation from the first tile (e.g., one or more (or all) tiles within a second separation; i.e., one or more (or all) tiles adjacent to the first tile or adjacent to a tile adjacent to the first tile). When vehicle 2702 moves to a second tile, vehicle 2702 can acquire the second tile. Vehicle 2702 can also acquire the first tile and one or more (or all) tiles adjacent to the second tile. Alternatively or additionally, vehicle 2702 may acquire one or more (or all) tiles within a predetermined distance from the second tile. Alternatively or additionally, vehicle 2702 may acquire one or more (or all) tiles within a predetermined separation from the second tile (e.g., one or more (or all) tiles within the second separation; i.e., one or more (or all) tiles adjacent to the second tile or adjacent to a tile adjacent to the second tile). In some embodiments, vehicle 2702 may also delete (or overwrite) the first tile and / or previously acquired tiles that are not adjacent to the second tile. Alternatively or additionally, vehicle 2702 may delete (or overwrite) one or more previously acquired tiles that are not within a predetermined distance from the second tile. Alternatively or additionally, vehicle 2702 may delete (or overwrite) one or more previously acquired tiles that are not within a predetermined separation from the second tile.

[0366] Figure 30 An exemplary procedure for obtaining one or more tiles is shown. Figure 30As shown, vehicle 2702 (and / or server 2701) can be configured to determine that at time point 1, vehicle 2702 is located in tile 5. Vehicle 2702 can also be configured to acquire (or load) adjacent tiles 1-4 and tiles 6-9. At time point 2, vehicle 2702 (and / or server 2701) can be configured to determine that vehicle 2702's location has moved from tile 5 to tile 3. Vehicle 2702 can be configured to acquire (or load) new tiles 10-14 adjacent to tile 3. Vehicle 2702 can also be configured to retain tiles 2, 3, 5, and 6, and delete tiles 1, 4, and 7-9. Therefore, vehicle 2702 can acquire (or load) a subset of tiles (e.g., 9 tiles) at a time to reduce memory usage and / or computational load. In some embodiments, vehicle 2702 can be configured to decode the tiles before loading data into memory.

[0367] Alternatively or additionally, vehicle 2702 may determine a sub-section of a tile (where the vehicle's position falls within that sub-section) and load tiles adjacent to that sub-section. For example, such as Figure 31A As shown, vehicle 2702 can determine its position within a sub-tile (the sub-tile with a dotted pattern in tile 5) and load map data from adjacent tiles (i.e., tiles 4, 7, and 9) into memory for processing. As another example, such as... Figure 31B As shown, vehicle 2702 can determine its position within the top-left sub-tile of tile 5. Vehicle 2702 can also load tiles adjacent to the top-left sub-tile of tile 5 (i.e., tiles 1, 2, and 4). Therefore, vehicle 2702 can acquire (or load) a subset of tiles (e.g., four tiles) at a time, reducing memory usage and / or computational load. As another example, such as... Figure 31C As shown, vehicle 2702 can determine its position within the upper right sub-tile and load the map data of adjacent tiles (i.e., tiles 2, 3, and 6) into memory for processing. As another example, such as... Figure 31D As shown, vehicle 2702 can determine that its position is in the lower right sub-tile and load the map data of the adjacent tiles (i.e., tiles 6, 8, and 9) into memory for processing. In some embodiments, vehicle 2702 can be configured to decode the tiles before loading the data into memory.

[0368] Figure 32This is a flowchart illustrating a process consistent with the disclosed embodiments for providing one or more map segments to one or more vehicles. One or more steps of process 3200 may be performed by a vehicle (e.g., vehicle 2702), a device associated with the main vehicle (e.g., vehicle device 2703), and / or a server (e.g., server 2701). Although the description of process 3200 provided below uses server 2701 as an example, those skilled in the art will understand that one or more steps of process 3200 may be performed by a vehicle (e.g., vehicle 2702) and vehicle devices (e.g., vehicle device 2703). For example, vehicle 2702 may determine a potential driving envelope based on navigation information. Along with or instead of receiving map data from server 2701, vehicle 2702 may also retrieve portions of map data related to the potential driving envelope from local storage and load the retrieved data into memory for processing.

[0369] In step 3201, navigation information can be received from the vehicle. For example, server 2701 can receive navigation information from vehicle 2702 via, for example, network 2705. In some embodiments, the navigation information received from the vehicle may include an indicator of vehicle position, an indicator of vehicle speed, and an indicator of vehicle direction of travel. For example, vehicle 2702 may be configured to receive data from one or more sensors, including, for example, a GPS device, a speed sensor, an accelerometer, a suspension sensor, etc., or combinations thereof. Vehicle 2702 may also be configured to determine navigation information such as vehicle position, speed, and / or driving direction based on the received data. Vehicle 2702 may also be configured to transmit navigation information to server 2701 via, for example, network 2705. Alternatively or additionally, vehicle 2702 may be configured to transmit sensor data to server 2701. Server 2701 may be configured to determine navigation information based on the received sensor data, which may include the position of vehicle 2702, the speed of vehicle 2702, and / or the direction of travel of vehicle 2702.

[0370] In some embodiments, vehicle 2702 may continuously transmit navigation information (and / or sensor data) to server 2701. Alternatively, vehicle 2702 may transmit navigation information (and / or sensor data) to server 2701 intermittently. For example, vehicle 2702 may transmit navigation information (and / or sensor data) to server 2701 multiple times over a period of time. For instance, vehicle 2702 may transmit navigation information (and / or sensor data) to server 2701 once per minute. Alternatively, vehicle 2702 may transmit navigation information when it has more reliable and / or faster network access (e.g., stronger wireless signal, via Wi-Fi connection, etc.).

[0371] In step 3202, the received navigation information can be analyzed, and the potential driving envelope of the vehicle can be determined. For example, server 2701 can analyze the position, speed, and / or driving direction of vehicle 2702 and determine the potential driving envelope (which may include a determined area relative to vehicle 2702). For example, such as Figure 28A As shown, server 2701 can determine the area covering the location of vehicle 2702, and determine the potential driving envelope with boundary 2811 based on the determined area.

[0372] In some embodiments, server 2701 may be configured to determine a potential driving envelope extending from and around the location of vehicle 2702. For example, as Figure 28A As shown, server 2701 can determine a line 2802 passing through the position (or centroid) of vehicle 2702. Server 2701 can also be configured to determine one side of the boundary of a potential driving envelope in the direction of travel of vehicle 2702, and the other side of the boundary of the potential driving envelope in the opposite direction of travel. For example, server 2701 can determine the upper boundary of the potential driving envelope in the direction of travel of vehicle 2702, and determine the lower boundary of the potential driving envelope in the opposite direction of travel of vehicle 2702. In some embodiments, the potential driving envelope may extend further along the direction of travel of the vehicle than in the opposite direction of travel. For example, as... Figure 28A As shown, the upper boundary of the potential driving envelope may be at a distance of 2803 from line 2802 (or a first distance from the position of vehicle 2702), and the lower boundary of the potential driving envelope may be at a distance of 2804 from line 2802 (or a second distance from the position of vehicle 2702). Distance 2803 may be greater than distance 2804 (and / or the first distance may be greater than the second distance). In some embodiments, the position of the centroid of the boundary may be offset from the position of vehicle 2702 along the driving direction of vehicle 2702.

[0373] Alternatively or additionally, when determining the potential travel envelope of vehicle 2702, server 2701 may consider the potential travel distance over a period of time (or time window). For example, server 2701 may determine the potential travel distance over a predetermined amount of time and determine a potential travel envelope that includes that potential travel distance. In some embodiments, the potential travel distance over the predetermined amount of time may be determined based on the position and / or speed of vehicle 2702. In some embodiments, server 2701 may further determine the potential travel envelope based on a selected or predetermined time window. The time window may be selected or determined based on an indicator of vehicle speed. The predetermined amount of time (or time window) may range from 0.1 seconds to 24 hours. In some embodiments, the predetermined amount of time (or time window) may be limited to sub-ranges of 0.1 seconds to 1 second, 1 second to 5 seconds, 5 to 10 seconds, 10 to 60 seconds, 1 minute to 5 minutes, 5 to 10 minutes, 10 to 60 minutes, 1 hour to 5 hours, 5 to 10 hours, and 10 to 24 hours. In some embodiments, a predetermined time period (or time window) can be determined based on the transmission frequency of navigation information from vehicle 2702 to server 2701. For example, server 2701 can determine the predetermined time period (or time window) based on the interval between two transmissions of navigation information from vehicle 2702. Server 2701 can determine a longer time period to determine the potential travel distance within the longer transmission interval.

[0374] In some embodiments, the potential driving envelope may include a boundary. The boundary of the potential driving envelope may have a shape including, but is not limited to, a triangle, a quadrilateral, a parallelogram, a rectangle, a square (or approximately square), a trapezoid, a rhombus, a hexagon, an octagon, a circle (or approximately circle), an ellipse, an egg, or any other shape or a combination thereof. Figures 28A-28D An exemplary potential driving envelope of a vehicle in region 2800, consistent with the disclosed embodiments, is shown. Figure 28A As shown, server 2701 can determine a potential driving envelope with boundary 2811 for vehicle 2702, which may include a trapezoidal shape. As another example, such as... Figure 28B As shown in Figure 28, server 2701 can determine a potential driving envelope for vehicle 2702 with boundary 2812, which may include an elliptical shape. As another example, as shown in Figure 28, server 2701 can determine a potential driving envelope for vehicle 2702 with boundary 2813, which may include a triangular shape. As another example, as shown in Figure 28, server 2701 can determine a potential driving envelope for vehicle 2702 with boundary 2814, which may include a rectangular shape.

[0375] In some embodiments, vehicle 2702 (and / or vehicle equipment 2703) may determine a potential driving envelope based on navigation information.

[0376] In step 3203, one or more map segments may be sent to vehicle 2702. In some embodiments, the map segments(s) may include map information of geographic areas that at least partially overlap with the potential driving envelope of vehicle 2702. For example, server 2701 may transmit one or more map segments via network 2705, which include map data of geographic areas that at least partially overlap with the potential driving envelope of vehicle 2702.

[0377] In some embodiments, one or more map segments include one or more tiles representing an area with a predetermined area. For example, such as Figure 28E As shown, server 2701 can identify one or more tiles 2831 that at least partially overlap with the potential driving envelope of vehicle 2702 (i.e., the potential driving envelope with boundary 2821), and transmit map data associated with tile 2831 to vehicle 2702 via network 2705.

[0378] In some embodiments, the area of ​​the tiles transmitted to vehicle 2702 can vary. For example, as... Figure 29A and Figure 29B As shown, an area (or map) can be divided into different levels, and tiles at specific levels can have specific areas. In some embodiments, the predetermined area of ​​a tile sent to vehicle 2702 can range from 0.25 square kilometers to 100 square kilometers, which can be limited to sub-ranges of 0.25 square kilometers to 1 square kilometer, 1 square kilometer to 10 square kilometers, 10 to 25 square kilometers, 25 to 50 square kilometers, and 50 to 100 square kilometers. In some embodiments, the predetermined area of ​​a tile can be less than or equal to 10 square kilometers. Alternatively, the predetermined area of ​​a tile can be less than or equal to 1 square kilometer. Alternatively, the predetermined area of ​​a tile can be less than or equal to 10 square kilometers. In some embodiments, a tile can have a rectangular shape, a square shape, a hexagonal shape, etc., or a combination thereof.

[0379] In some embodiments, the map information sent to vehicle 2702 may include a polynomial representation of a target trajectory along one or more road segments, as described elsewhere in this disclosure. For example, the map information may include... Figure 9A , Figure 9B and Figure 11A The disclosed embodiments illustrate a polynomial representation of a portion of a road segment. For example, map information may include a polynomial representation of a target trajectory determined based on two or more reconstructed trajectories previously traversed by a vehicle along one or more road segments.

[0380] In some embodiments, after receiving one or more road segments, vehicle 2702 may navigate based on the one or more road segments as described elsewhere in this disclosure. For example, vehicle 2702 may be configured to perform one or more navigation actions (e.g., turning, stopping at a location, etc.) based on the received one or more road segments. Alternatively or additionally, vehicle 2702 may be configured to perform one or more navigation actions based on a polynomial representation of a target trajectory along one or more road segments.

[0381] In some embodiments, vehicle 2702 may receive one or more road segments and store them in a storage device. Vehicle 2702 may also load one or more tiles contained in the one or more road segments into memory for processing. For example, as... Figure 30 As shown, vehicle 2702 (and / or server 2701) can be configured to determine that at time point 1, vehicle 2702's position is in tile 5. Vehicle 2702 can also be configured to acquire (or load) adjacent tiles 1-4 and 6-9. At time point 2, vehicle 2702 (and / or server 2701) can be configured to determine that vehicle 2702's position has moved from tile 5 to tile 3. Vehicle 2702 can be configured to acquire (or load) new tiles 10-14, which are adjacent to tile 3. Vehicle 2702 can also be configured to retain tiles 2, 3, 5, and 6, and delete tiles 1, 4, and 7-9.

[0382] Alternatively or additionally, vehicle 2702 may determine a sub-section of a tile (where the vehicle's position falls within that sub-section) and load tiles adjacent to that sub-section. For example, such as Figure 31A As shown, vehicle 2702 can determine its position within a sub-tile (the sub-tile with a dotted pattern in tile 5) and load map data from adjacent tiles (i.e., tiles 4, 7, and 9) into memory for processing. As another example, such as... Figure 31B As shown, vehicle 2702 can determine that its position is in the upper-left sub-tile of tile 5. Vehicle 2702 can also load tiles adjacent to the upper-left sub-tile of tile 5 (i.e., tiles 1, 2, and 4). In some embodiments, vehicle 2702 can be configured to decode tiles before loading data into memory.

[0383] In some embodiments, vehicle 2702 may obtain one or more road segments from local storage, rather than receiving one or more road segments from server 2701 via network 2705. For example, vehicle 2702 may determine a potential driving envelope based on analysis of navigation information and determine one or more road segments including map information of geographic areas that at least partially overlap with the potential driving envelope of vehicle 2702. Vehicle 2702 may also obtain data for one or more road segments from local storage. In some embodiments, vehicle 2702 may load the data for one or more road segments into its memory for processing.

[0384] Bandwidth management for map generation and refinement

[0385] As described elsewhere in this disclosure, utilizing and interpreting the large amounts of data collected by vehicles (e.g., captured image data, map data, GPS data, sensor data, etc.) presents several design challenges. For example, it may be necessary to upload the data collected by the vehicle to a server. The large amount of data to be uploaded can easily deplete or hinder the vehicle's transmission bandwidth. Furthermore, it can be challenging for the server to analyze new data to update relevant parts of the map based on the new data. In addition, different density levels can be used to map different types of features. For example, mapping non-semantic features may require a density level of 330 kB per kilometer, compared to approximately 20 kB per kilometer for semantic features. Considering the computational resources required to collect non-semantic feature information and certain hard-limited bandwidth caps that may be imposed on each vehicle (e.g., 100 MB per year), there may not be sufficient resources for vehicles to continuously collect non-semantic feature information.

[0386] This system and method enable control not only over whether driving data is collected, but also over when and how it is collected. For example, the disclosed system and method allow a server to determine whether a primary vehicle is entering an area containing a region of interest. Upon confirming that the vehicle has entered the area, the server can cause the vehicle to begin collecting higher-density non-semantic feature information. If it is determined that the vehicle has driven through a point of interest, the server can cause the vehicle to upload the collected non-semantic feature information. If it is determined that the vehicle has not driven through the region of interest, the server can cause the primary vehicle to discard the collected non-semantic feature information. The disclosed system and method also enable the server to update the map based on the non-semantic feature information collected by the vehicle.

[0387] Figure 33 An exemplary system for automatically generating navigation maps associated with one or more road segments, consistent with embodiments of this disclosure, is shown. Figure 30As shown, system 3300 may include server 3301, one or more vehicles 3302 and one or more vehicle devices 3303 associated with the vehicles, database 3304, and network 3305. For example, vehicle 3302 and / or vehicle device 3303 may be configured to collect first navigation information associated with the environment traversed by vehicle 3302 at a first density level when vehicle 3302 is traveling at a predetermined distance from a geographic area of ​​interest. Vehicle 3302 and / or vehicle device 3303 may also be configured to collect second navigation information associated with the environment traversed by vehicle 3302 at a second density level, which may be greater than the first density level, when vehicle 3302 is traveling at or within a predetermined distance from the geographic area of ​​interest.

[0388] Server 3301 can be configured to receive first navigation information and / or second navigation information associated with the environment traversed by vehicle 3302. Database 3304 can be configured to store information for components of system 3300 (e.g., server 3301, vehicle 3302, and / or vehicle equipment 3303). Network 3305 can be configured to facilitate communication between components of system 3300.

[0389] Server 3301 can be configured to initiate the collection of first navigation information associated with the environment traversed by vehicle 3302. The first navigation information can be collected at a first density level. Server 3301 can also be configured to determine the position of vehicle 3302 based on output associated with GPS sensors associated with vehicle 3302. Server 3301 can also be configured to determine whether vehicle 3302 is located at or within a predetermined distance from a geographic area of ​​interest (or its boundary). Server 3301 can also be configured to initiate the collection of second navigation information associated with the environment traversed by vehicle 3302 based on the determination that vehicle 3302 is located at or within a predetermined distance from a geographic area of ​​interest. The second navigation information can be collected at a second density level, which may be greater than the first density level. Server 3301 can also be configured to cause vehicle 3302 to upload at least one of the collected first navigation information or the collected second navigation information (or a portion thereof). Server 3301 can also be configured to update a navigation map based on the uploaded collected first navigation information or the collected second navigation information. In some embodiments, server 3301 may be a cloud server performing the functions disclosed herein. The term "cloud server" refers to a computer platform that provides services via a network (e.g., the Internet). In this example configuration, server 3301 may use virtual machines that do not correspond to separate hardware. For example, computing and / or storage capabilities can be implemented by allocating appropriate portions of computing / storage capabi...

Claims

1. A system for navigating a vehicle, the system comprising: At least one processor includes circuitry and memory, wherein the memory includes instructions that, when executed by the circuitry, cause the at least one processor to: Receive navigation information associated with the vehicle, the navigation information including at least an indicator of the vehicle's location; The method involves determining multiple target navigation map segments to be retrieved from a map database, the map database including multiple stored navigation map segments, each corresponding to a real-world region, wherein the determination of the multiple target navigation map segments is based on an indicator of vehicle location and on map segment connectivity information associated with the multiple stored navigation map segments, the map segment connectivity information indicating whether one or more boundaries between the multiple stored navigation map segments can be crossed by a vehicle traveling along at least one road segment, wherein determining the multiple target navigation map segments to be retrieved includes based on the connectivity between the current navigation map segment and one or more navigation map segments adjacent to the current navigation map segment as indicated by the map segment connectivity information, excluding the one or more navigation map segments adjacent to the current navigation map segment where the vehicle is located; Initiate the download of the multiple target navigation map segments from the map database; and The vehicle is navigated along at least one target trajectory contained in one or more of the plurality of target navigation map segments downloaded from the map database.

2. The system according to claim 1, wherein, The map database is used for remote positioning of the vehicle.

3. The system according to claim 1, wherein, If the map segment connectivity information indicates that the vehicle cannot directly navigate from the current navigation map segment in which the vehicle is located to one or more navigation map segments adjacent to the current navigation map segment, then the multiple target navigation map segments to be obtained exclude the one or more navigation map segments adjacent to the current navigation map segment.

4. The system according to claim 1, wherein, The memory includes instructions that, when executed by the circuit, cause the at least one processor to determine the plurality of target navigation map segments to be acquired based on at least one of an indicator of the vehicle's speed or an indicator of the vehicle's direction of travel.

5. The system according to claim 1, wherein, The memory includes instructions that, when executed by the circuit, cause the at least one processor to prioritize the plurality of target navigation map segments based on the map segment connectivity information.

6. The system according to claim 5, wherein, Based on the indication of the map segment connectivity information, the first navigation map segment is assigned a higher priority than the second navigation map segment, the indication being that the navigable connection of the first navigation map segment relative to the current navigation map segment where the vehicle is located is shorter than the distance of the navigable connection between the second navigation map segment and the current navigation map segment.

7. The system according to claim 6, wherein, The first navigation map segment does not share a boundary with the current navigation map segment, but the second navigation map segment shares a boundary with the current navigation map segment.

8. The system according to claim 1, wherein, Each of the plurality of stored navigation map segments includes a field that stores map segment connectivity information.

9. The system according to claim 8, wherein, The memory includes instructions that, when executed by the circuit, cause the at least one processor to prioritize the plurality of target navigation map segments by accessing at least two fields of the plurality of stored navigation map segments and analyzing the map segment connectivity information stored in the accessed fields before acquiring one or more of the plurality of stored map segments.

10. The system according to claim 8, wherein, The accessed fields are associated with a predetermined number of map segments near the current navigation map segment where the vehicle is located.

11. The system according to claim 10, wherein, The map segments are arranged into tiles, and the accessed fields are associated with eight tiles surrounding the central tile where the vehicle is located.

12. The system according to claim 10, wherein, The map segments are arranged into tiles, and the accessed fields are associated with 24 tiles surrounding the central tile where the vehicle is located.

13. The system according to claim 9, wherein, The accessed field is selected based on at least one of the vehicle's direction of travel or the vehicle's speed.

14. The system according to claim 1, wherein, The map segment connectivity information takes into account the available driving directions that cross the boundary of the navigation map segment along the road segment.

15. The system according to claim 1, wherein, The map segment connectivity information is stored as boolean bits associated with one or more boundaries, which are associated with each of the plurality of navigation map segments.

16. The system according to claim 1, wherein, Each of the plurality of stored navigation map segments includes a plurality of map sub-segments, and each of the plurality of map sub-segments includes map segment connectivity information specific to that map sub-segment.

17. The system according to claim 1, wherein, The memory includes instructions that, when executed by the circuit, cause the at least one processor to periodically update the plurality of target navigation map segments to be retrieved from the map database.

18. The system according to claim 17, wherein, The periodic updates occur at a rate of more than once per second.

19. The system according to claim 17, wherein, The periodic update occurs before all the identified target navigation map segments have been retrieved from the map database.

20. The system according to claim 1, wherein, The vehicle's position indicator is accurate to within 5 centimeters of the vehicle's actual position.

21. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations, the operations comprising: Receive navigation information associated with the vehicle, the navigation information including at least an indicator of the vehicle's location; The method involves determining multiple target navigation map segments to be retrieved from a map database, the map database including multiple stored navigation map segments, each corresponding to a real-world region, wherein the determination of the multiple target navigation map segments is based on an indicator of vehicle location and on map segment connectivity information associated with the multiple stored navigation map segments, the map segment connectivity information indicating whether one or more boundaries between the multiple stored navigation map segments can be crossed by a vehicle traveling along at least one road segment, wherein determining the multiple target navigation map segments to be retrieved includes based on the connectivity between the current navigation map segment and one or more navigation map segments adjacent to the current navigation map segment as indicated by the map segment connectivity information, excluding the one or more navigation map segments adjacent to the current navigation map segment where the vehicle is located; Initiate the download of the multiple target navigation map segments from the map database; and The vehicle is navigated along at least one target trajectory contained in one or more of the plurality of target navigation map segments downloaded from the map database.

22. The non-transitory computer-readable medium according to claim 21, wherein, The map database is used for remote positioning of the vehicle.

23. The non-transitory computer-readable medium according to claim 21, wherein, The map segment connectivity information takes into account the available driving directions that cross the boundary of the navigation map segment along the road segment.

24. The non-transitory computer-readable medium according to claim 21, wherein, The map segment connectivity information is stored as boolean bits associated with one or more boundaries, which are associated with each of the plurality of navigation map segments.

25. The non-transitory computer-readable medium according to claim 21, wherein, Each of the plurality of stored navigation map segments includes a plurality of map sub-segments, and each of the plurality of map sub-segments includes map segment connectivity information specific to that map sub-segment.

26. The non-transitory computer-readable medium according to claim 21, wherein, When the instruction is executed by the at least one processor, it causes the at least one processor to periodically update the plurality of target navigation map segments to be obtained from the map database.

27. The non-transitory computer-readable medium according to claim 26, wherein, The periodic updates occur at a rate of more than once per second.

28. The non-transitory computer-readable medium according to claim 26, wherein, The periodic update occurs before all the identified target navigation map segments have been retrieved from the map database.

29. A non-transitory computer-readable medium comprising a digital map for vehicle navigation, the digital map comprising: Multiple navigation map segments, each corresponding to a real-world region; as well as For each of the plurality of navigation map segments: at least one map segment connectivity indicator associated with each boundary shared with adjacent navigation map segments, wherein the at least one map segment connectivity indicator is stored as a Boolean bit on the computer-readable medium, the at least one map segment connectivity indicator indicating whether the boundary can be crossed by a vehicle traveling along at least one road segment.

30. The non-transitory computer-readable medium according to claim 29, wherein, The at least one map segment connectivity indicator associated with each boundary shared with adjacent navigation map segments indicates a unidirectional boundary.

31. The non-transitory computer-readable medium according to claim 29, wherein, The at least one map segment connectivity indicator associated with each boundary shared with adjacent navigation map segments indicates a bidirectional boundary.

32. The non-transitory computer-readable medium according to claim 29, wherein, The digital map includes a representation of the target trajectory that a vehicle navigation system should follow when traversing road segments represented in the digital map.

33. The non-transitory computer-readable medium according to claim 32, wherein, The digital map includes representations of identified landmarks for use by vehicle navigation systems to locate the vehicle's position relative to at least one of the target trajectories.

34. A device for navigating a vehicle, the device comprising: A component for receiving navigation information associated with the vehicle, the navigation information including at least an indicator of the vehicle's location; A component for determining multiple target navigation map segments to be retrieved from a map database, the map database including multiple stored navigation map segments, each stored navigation map segment corresponding to a real-world region, wherein the determination of the multiple target navigation map segments is based on an indicator of vehicle location and on map segment connectivity information associated with the multiple stored navigation map segments, the map segment connectivity information indicating whether one or more boundaries between the multiple stored navigation map segments can be crossed by a vehicle traveling along at least one road segment, wherein the component for determining the multiple target navigation map segments to be retrieved from the map database excludes the one or more navigation map segments adjacent to the current navigation map segment where the vehicle is located based on the connectivity between the current navigation map segment and one or more navigation map segments adjacent to the current navigation map segment indicated by the map segment connectivity information; A component for initiating the download of the plurality of target navigation map segments from the map database; and A component for navigating the vehicle along a trajectory of at least one target contained in one or more of the plurality of target navigation map segments downloaded from the map database.

35. The device according to claim 34, wherein, The map database is used for remote positioning of the vehicle.

36. The device according to claim 34, wherein, If the map segment connectivity information indicates that the vehicle cannot directly navigate from the current navigation map segment in which the vehicle is located to one or more navigation map segments adjacent to the current navigation map segment, then the multiple target navigation map segments to be obtained exclude the one or more navigation map segments adjacent to the current navigation map segment.

37. The device according to claim 34, wherein, The device further includes a component for determining the plurality of target navigation map segments to be acquired based on at least one of an indicator of the vehicle's speed or an indicator of the vehicle's direction of travel.

38. The device according to claim 34, wherein, The device further includes a component for prioritizing the plurality of target navigation map segments based on the map segment connectivity information.

Images