Aligning road information for navigation

Abstract

The invention relates to a system and method for aligning navigation information from multiple vehicles. In one embodiment, at least one processing device may receive first navigation information from a first vehicle and second navigation information from a second vehicle. The first navigation information and the second navigation information may be associated with a public road segment. The processor may divide a common road segment into a first road section and a second road section connected at a common point. The processor may then align the first navigation information and the second navigation information by rotating at least a portion of the first navigation information or the second navigation information relative to the common point. The processor may store the aligned navigation information in association with the public road segment and transmit the aligned navigation information to the vehicle for navigation along the public road segment.

Description

[0001] This patent application is a divisional application of the following invention patent applications:

[0002] Application number: 202080019111.0

[0003] Application date: March 6, 2020

[0004] Invention Title: Aligning road information for navigation

[0005] CROSS-REFERENCE TO RELATED APPLICATIONS

[0006] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 815,045, filed on March 7, 2019, and U.S. Provisional Patent Application No. 62 / 874,711, filed on July 16, 2019. The entireties of the above applications are incorporated herein by reference in their entirety. Technical Field

[0007] The present disclosure relates generally to autonomous vehicle navigation. Background Art

[0008] As technology continues to advance, the goal of fully autonomous vehicles that can navigate on roads is imminent. Autonomous vehicles may need to consider a variety of factors and make appropriate decisions based on those factors to safely and accurately reach the intended destination. For example, autonomous vehicles may need to process and interpret visual information (e.g., information captured from a camera), information from radar or lidar, and may also use information obtained from other sources (e.g., from a GPS device, a speed sensor, an accelerometer, a suspension sensor, etc.). At the same time, in order to navigate to the destination, the autonomous vehicle may also need to identify its location within a specific road (e.g., a specific lane in a multi-lane road), navigate side by side with other vehicles, avoid obstacles and pedestrians, observe traffic signals and signs, and drive from one road to another at an appropriate intersection or junction. Harnessing and interpreting the large amount of information collected by the autonomous vehicle when it travels to its destination creates many design challenges. The massive amount of data (e.g., captured image data, map data, GPS data, sensor data, etc.) that the autonomous vehicle may need to analyze, access and / or store creates challenges that may actually limit or even adversely affect autonomous navigation. Furthermore, if autonomous vehicles rely on traditional mapping technologies to navigate, the massive amounts of data required to store and update maps will pose a huge challenge.

[0009] In addition to collecting data for updating maps, autonomous vehicles must be able to use maps for navigation. Therefore, the size and detail of the maps, as well as their construction and transmission, must be optimized. Furthermore, autonomous vehicles must navigate using maps as well as constraints based on the vehicle's surroundings to ensure the safety of its passengers and other drivers and pedestrians on the road. Summary of the invention

[0010] Embodiments consistent with the present disclosure provide systems and methods for autonomous vehicle navigation. The disclosed embodiments may use cameras to provide vehicle navigation features. For example, consistent with the disclosed embodiments, the disclosed system may include one, two or more cameras that monitor the environment of the vehicle. The disclosed system may provide navigation responses based on, for example, analysis of images captured by one or more cameras. The disclosed system may also provide the construction of crowdsourced sparse maps and navigation using crowdsourced sparse maps. Other disclosed systems may use correlation analysis of images to perform positioning, which may supplement navigation performed using sparse maps. The navigation response may also take other data into account, including, for example, global positioning system (GPS) data, sensor data (e.g., from accelerometers, velocity sensors, suspension sensors, etc.) and / or other map data. Finally, the disclosed embodiments may use comfort constraints and safety constraints to fuse data from multiple sources such as cameras, sensors, maps, etc., so as to optimize vehicle navigation without endangering other drivers and pedestrians.

[0011] In one embodiment, a computer-implemented method for aligning navigation information from multiple vehicles includes: receiving first navigation information from a first vehicle; receiving second navigation information from a second vehicle, wherein the first navigation information and the second navigation information are associated with a road segment; dividing the first navigation information into at least a first portion and a second portion, and dividing the second navigation information into at least a first portion and a second portion, wherein the first portion of the first navigation information and the first portion of the second navigation information represent a first common section of the road segment, and wherein the second portion of the first navigation information and the second portion of the second navigation information represent a second common section of the road segment; aligning the first portion of the first navigation information with the first portion of the second navigation information, and aligning the second portion of the first navigation information with the second portion of the second navigation information; generating a road model based on the aligned portions; and sending at least a portion of the road model to one or more vehicles for navigating the one or more vehicles along the road segment; wherein the first navigation information includes a first position of at least one landmark based on one or more images captured by a camera included in the first vehicle, and the second navigation information includes a second position of the at least one landmark based on one or more images captured by a camera included in the second vehicle.

[0012] In one embodiment, a non-transitory computer-readable medium may store instructions that, when executed by at least one processor, cause the at least one processor to receive first navigation information from a first vehicle and second navigation information from a second vehicle, wherein the first navigation information and the second navigation information are associated with a public road segment. The at least one processor may also divide the public road segment into at least a first road segment and a second road segment, wherein the first road segment and the second road segment are connected at a common point, and align the first navigation information with the second navigation information relative to the public road segment. Aligning the first navigation information with the second navigation information may include rotating at least a portion of the first navigation information or at least a portion of the second navigation information relative to the common point. The at least one processor may also store the aligned navigation information in association with the public road segment, and transmit at least a portion of the aligned navigation information to one or more vehicles for navigating the one or more vehicles along the public road segment.

[0013] In one embodiment, a server for aligning navigation information from multiple vehicles may include at least one processor. The at least one processor may be configured to receive first navigation information from a first vehicle and second navigation information from a second vehicle, wherein the first navigation information and the second navigation information are associated with a public road segment. The at least one processor may also divide the public road segment into at least a first road segment and a second road segment, wherein the first road segment and the second road segment are connected at a common point, and align the first navigation information with the second navigation information relative to the public road segment. Aligning the first navigation information with the second navigation information may include rotating at least a portion of the first navigation information or at least a portion of the second navigation information relative to the common point. The at least one processor may also be configured to store the aligned navigation information in association with the public road segment, and transmit at least a portion of the aligned navigation information to one or more vehicles for navigating the one or more vehicles along the public road segment.

[0014] In one embodiment, a computer-implemented method for aligning navigation information from multiple vehicles may include receiving first navigation information from a first vehicle and receiving second navigation information from a second vehicle, wherein the first navigation information and the second navigation information are associated with a common road segment. The method may include dividing the common road segment into at least a first road segment and a second road segment, wherein the first road segment and the second road segment are connected at a common point, and aligning the first navigation information with the second navigation information relative to the common road segment. Aligning the first navigation information with the second navigation information may include rotating at least a portion of the first navigation information or at least a portion of the second navigation information relative to the common point. The method may also include storing the aligned navigation information in association with the common road segment, and transmitting at least a portion of the aligned navigation information to one or more vehicles for navigating the one or more vehicles along the common road segment.

[0015] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium may store program instructions that are executed by at least one processing device and may perform any of the methods described herein.

[0016] The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. In the drawings:

[0018] Figure 1 are pictorial representations of exemplary systems consistent with the disclosed embodiments.

[0019] Figure 2Ais a diagrammatic side view representation of an exemplary vehicle incorporating systems consistent with the disclosed embodiments.

[0020] Figure 2B yes Figure 2A A diagrammatic top view representation of a vehicle and systems consistent with the disclosed embodiments is shown in FIG.

[0021] Figure 2C is a diagrammatic top view representation of another embodiment of a vehicle incorporating a system consistent with the disclosed embodiments.

[0022] Figure 2D is a diagrammatic top view representation of yet another embodiment of a vehicle incorporating a system consistent with the disclosed embodiments.

[0023] Figure 2E is a diagrammatic top view representation of yet another embodiment of a vehicle incorporating a system consistent with the disclosed embodiments.

[0024] Figure 2F is a pictorial representation of an exemplary vehicle control system consistent with the disclosed embodiments.

[0025] Figure 3A is a pictorial representation of the interior of a vehicle including a rearview mirror and a user interface for a vehicle imaging system consistent with the disclosed embodiments.

[0026] Figure 3B is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against a vehicle windshield consistent with the disclosed embodiments.

[0027] Figure 3C yes Figure 3B 00140 10 ...

[0028] Figure 3D is an illustration of an example of a camera mount configured to be positioned behind a rearview mirror and against a vehicle windshield consistent with the disclosed embodiments.

[0029] Figure 4 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations consistent with the disclosed embodiments.

[0030] Figure 5A is a flow chart illustrating an exemplary process for eliciting one or more navigation responses based on monocular image analysis consistent with the disclosed embodiments.

[0031] Figure 5B is a flow chart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a set of images consistent with the disclosed embodiments.

[0032] Figure 5C is a flow chart illustrating an exemplary process for detecting road signs and / or lane geometry information in a set of images consistent with the disclosed embodiments.

[0033] Figure 5D is a flow chart illustrating an exemplary process for detecting traffic lights in a set of images consistent with the disclosed embodiments.

[0034] Figure 5E is a flow chart illustrating an exemplary process for inducing one or more navigation responses based on a vehicle path, consistent with the disclosed embodiments.

[0035] Fig. 5F is a flow chart illustrating an exemplary process for determining whether a leading vehicle is changing lanes, consistent with the disclosed embodiments.

[0036] Figure 6 is a flow chart illustrating an exemplary process for eliciting one or more navigation responses based on stereoscopic image analysis consistent with the disclosed embodiments.

[0037] Figure 7 is a flow chart illustrating an exemplary process for eliciting one or more navigation responses based on analysis of three sets of images consistent with the disclosed embodiments.

[0038] Fig. 8A A polynomial representation of a portion of a road segment consistent with the disclosed embodiments is shown.

[0039] Figure 8B A curve representing a target trajectory of a vehicle for a particular road segment in three-dimensional space included in a sparse map consistent with the disclosed embodiments is shown.

[0040] Fig.9A A polynomial representation of a trajectory consistent with the disclosed embodiments is shown.

[0041] Fig. 9B and Fig. 9C A target trajectory along a multi-lane road is shown consistent with the disclosed embodiments.

[0042] Fig.9D Example road signature profiles consistent with disclosed embodiments are shown.

[0043] Fig.10 An example autonomous vehicle road navigation model represented by a plurality of three-dimensional splines is shown consistent with the disclosed embodiments.

[0044] Fig.11A map skeleton generated from combining positioning information from multiple drives consistent with the disclosed embodiments is shown.

[0045] Fig.12 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations consistent with the disclosed embodiments.

[0046] Fig.13A and Fig. 13B A graphical description of example safety constraints and comfort constraints consistent with the disclosed embodiments is provided.

[0047] Fig. 13C and Fig.13D A graphical description of further example safety constraints and comfort constraints consistent with the disclosed embodiments is provided.

[0048] Fig.14 is a flow chart of an exemplary process for navigating a host vehicle based on safety constraints and comfort constraints consistent with the disclosed embodiments.

[0049] Fig.15 is an exemplary block diagram configured to store instructions for performing one or more operations consistent with the disclosed embodiments.

[0050] Fig.16 Example road data generated by combining navigation information from many drives and an example global map generated by combining the road data are shown consistent with the disclosed embodiments.

[0051] Fig.17 is a flow chart illustrating an exemplary process for aligning navigation information from multiple vehicles consistent with the disclosed embodiments.

[0052] Fig.18A Example vehicle data that may be aligned consistent with the disclosed embodiments is shown.

[0053] Fig.18B The errors that may be introduced when aligning driving over larger distances consistent with the disclosed embodiments are shown.

[0054] Fig.19A A number of segments are shown that may be used for alignment of road information consistent with the disclosed embodiments.

[0055] Fig.19B The resulting alignment based on multiple segments is shown consistent with the disclosed embodiments.

[0056] Fig. 20 is a flow chart representing an example process for aligning navigation information from multiple vehicles consistent with the disclosed embodiments. DETAILED DESCRIPTION

[0057] The following detailed description refers to the accompanying drawings. Whenever possible, the same reference numerals are used in the drawings and the following description to refer to the same or similar parts. Although several illustrative embodiments are described herein, modifications, adaptations and other implementations are possible. For example, the components shown in the drawings may be replaced, added or modified, and the illustrative methods described herein may be modified by replacing, reordering, removing or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the appropriate scope is defined by the appended claims.

[0058] Autonomous Vehicle Overview

[0059] As used throughout this disclosure, the term "autonomous vehicle" refers to a vehicle that is capable of implementing at least one navigation change without driver input. "Navigation change" refers to one or more changes in the steering, braking, or acceleration of the vehicle. In order to achieve autonomy, the vehicle does not need to be fully automatic (e.g., fully operable without a driver or without driver input). Instead, an autonomous vehicle includes vehicles that can operate under the control of a driver during certain time periods and operate without driver control during other time periods. An autonomous vehicle may also include a vehicle that controls only some aspects of vehicle navigation, such as steering (e.g., maintaining a vehicle route between vehicle lane limits), but other aspects may be left to the driver (e.g., braking). In some cases, an autonomous vehicle may handle some or all aspects of braking, speed control, and / or steering of the vehicle.

[0060] Since human drivers typically rely on visual cues and observation sequences in order to control vehicles, traffic infrastructure is built accordingly, where lane markings, traffic signs, and traffic lights are all designed to provide visual information to drivers. In view of these design characteristics of the traffic infrastructure, an autonomous vehicle may include a camera and a processing unit that analyzes visual information captured from the vehicle's environment. The visual information may include, for example, components of the traffic infrastructure (e.g., lane markings, traffic signs, traffic lights, etc.) that can be observed by the driver, as well as other obstacles (e.g., other vehicles, pedestrians, debris, etc.). In addition, the autonomous vehicle may also use stored information, such as information that provides a model of the vehicle's environment when navigating. For example, a vehicle may use GPS data, sensor data (e.g., from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data to provide information related to its environment while the vehicle is traveling, and the vehicle (and other vehicles) may use the information to locate itself on the model.

[0061] In some embodiments of the present disclosure, an autonomous vehicle may use information obtained while navigating (from cameras, GPS devices, accelerometers, velocity sensors, suspension sensors, etc.). In other embodiments, an autonomous vehicle may use information obtained from past navigations of the vehicle (or other vehicles) while navigating. In other embodiments, an autonomous vehicle may use a combination of information obtained while navigating and information obtained from past navigations. The following sections provide an overview of systems consistent with the disclosed embodiments, followed by an overview of forward imaging systems and methods consistent with the systems. The following sections disclose systems and methods for building, using, and updating sparse maps for autonomous vehicle navigation.

[0062] System Overview

[0063] Figure 1 1 is a block diagram of a representation system 100 consistent with the disclosed exemplary embodiments. The system 100 may include various components depending on the requirements of a particular implementation. In some embodiments, the system 100 may include a processing unit 110, an image acquisition unit 120, a position sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. The processing unit 110 may include one or more processing devices. In some embodiments, the processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, the image acquisition unit 120 may include any number of image acquisition devices and components depending on the requirements of a particular application. In some embodiments, the image acquisition unit 120 may include one or more image capture devices (e.g., cameras), such as image capture device 122, image capture device 124, and image capture device 126. The system 100 may also communicatively connect the processing device 110 to a data interface 128 of the image acquisition device 120. For example, the data interface 128 may include any wired and / or wireless link or links for transmitting image data acquired by the image acquisition device 120 to the processing unit 110 .

[0064] The wireless transceiver 172 may include one or more devices configured to exchange transmissions over an air interface to one or more networks (e.g., cellular, Internet, etc.) using radio frequencies, infrared frequencies, magnetic fields, or electric fields. The wireless transceiver 172 may transmit and / or receive data using any known standard (e.g., Wi-Fi, Bluetooth Smart, 802.15.4, ZigBee, etc.). Such transmissions may include communications from the host vehicle to one or more remotely located servers. Such transmissions may also include communications (one-way or two-way) between the host vehicle and one or more target vehicles in the host vehicle's environment (e.g., to coordinate navigation of the host vehicle with respect to or in conjunction with target vehicles in the host vehicle's environment), or even broadcast transmissions to unspecified recipients in the vicinity of the transmitting vehicle.

[0065] Both application processor 180 and image processor 190 may include various types of hardware-based processing devices. For example, either or both of application processor 180 and image processor 190 may include a microprocessor, a preprocessor (such as an image preprocessor), a graphics processor, a central processing unit (CPU), support circuits, a digital signal processor, an integrated circuit, a memory, or any other type of device suitable for running applications and suitable for image processing and analysis. In some embodiments, application processor 180 and / or image processor 190 may include any type of single-core or multi-core processor, a mobile device microcontroller, a central processing unit, etc. Various processing devices may be used, including, for example, a processor such as a processor that can be used to process images. processors from manufacturers such as GPUs from manufacturers such as wait).

[0066] In some embodiments, the application processor 180 and / or the image processor 190 may include a processor that can be Any of the EyeQ series processor chips available. These processor designs each contain multiple processing units with local memory and instruction sets. Such processors may contain video inputs for receiving image data from multiple image sensors, and may also contain video output capabilities. In one example, Using 90 nanometer-micron technology operating at 332Mhz. The architecture consists of two floating-point hyperthreaded 32-bit RISC CPUs ( cores), five visual computing engines (VCEs), and three vector microcode processors The MIPS34K CPU manages the five VCEs, three VMPs, and a 64-bit mobile DDR controller from Denali, a 128-bit internal Sonics Interconnect, dual 16-bit video input and 18-bit video output controllers, a 16-channel DMA, and several peripherals. TMand DMA, a second MIPS34K CPU and multi-channel DMA and other peripherals. These five VCEs, three and MIPS34K CPUs can perform the intensive visual calculations required for multi-function bundled applications. In another example, a third generation processor and a processor larger than 100nm can be used in the disclosed embodiments. Six times stronger In other examples, the disclosed embodiments may be used and / or Of course, any updated or future EyeQ processing devices may also be used with the disclosed embodiments.

[0067] Any of the processing devices disclosed herein may be configured to perform certain functions. Configuring a processing device (such as any described EyeQ processor or other controller or microprocessor) to perform certain functions may include programming computer executable instructions and making these instructions available to the processing device for execution during operation of the processing device. In some embodiments, configuring the processing device may include programming the processing device directly with architectural instructions. For example, a processing device such as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc. may be configured using, for example, one or more hardware description languages ​​(HDLs).

[0068] In other embodiments, configuring the processing device may include storing executable instructions on a memory accessible to the processing device during operation. For example, the processing device may access the memory during operation to obtain and execute the stored instructions. In either case, the processing device configured to perform the sensing, image analysis, and / or navigation functions disclosed herein represents a dedicated hardware-based system that controls multiple hardware-based components of the host vehicle.

[0069] Although Figure 1 Two separate processing devices included in processing unit 110 are depicted, but more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to complete the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by more than two processing devices. Furthermore, in some embodiments, system 100 may include one or more of processing units 110 without including other components (such as image acquisition unit 120).

[0070] The processing unit 110 may include various types of devices. For example, the processing unit 110 may include various devices, such as a controller, an image preprocessor, a central processing unit (CPU), a graphics processing unit (GPU), support circuits, a digital signal processor, an integrated circuit, a memory, or any other type of device for image processing and analysis. The image preprocessor may include a video processor for capturing, digitizing, and processing images from an image sensor. The CPU may include any number of microcontrollers or microprocessors. The GPU may also include any number of microcontrollers or microprocessors. The support circuit may be any number of circuits known in the art, including caches, power supplies, clocks, and input-output circuits. The memory may store software that controls the operation of the system when executed by the processor. The memory may include a database and image processing software. The memory may include any number of random access memories, read-only memories, flash memories, disk drives, optical storage devices, tape storage devices, removable storage devices, and other types of storage devices. In one example, the memory may be separate from the processing unit 110. In another example, the memory may be integrated into the processing unit 110.

[0071] Each memory 140, 150 may contain software instructions that, when executed by a processor (e.g., application processor 180 and / or image processor 190), may control the operation of various aspects of system 100. These memory units may contain various databases and image processing software, as well as trained systems, such as neural networks or, for example, deep neural networks. The memory units may contain random access memory (RAM), read-only memory (ROM), flash memory, disk drives, optical storage devices, tape storage devices, removable storage devices, and / or any other type of storage device. In some embodiments, the memory units 140, 150 may be separate from the application processor 180 and / or image processor 190. In other embodiments, these memory units may be integrated into the application processor 180 and / or image processor 190.

[0072] Position sensor 130 may include any type of device suitable for determining a location associated with at least one component of system 100. In some embodiments, position sensor 130 may include a GPS receiver. Such a receiver may determine user location and velocity by processing signals broadcast by global positioning system satellites. Position information from position sensor 130 may be made available to application processor 180 and / or image processor 190.

[0073] In some embodiments, system 100 may include components such as a speed sensor (eg, tachometer, speedometer) for measuring the speed of vehicle 200 and / or an accelerometer (single-axis or multi-axis) for measuring the acceleration of vehicle 200 .

[0074] The user interface 170 may include any device suitable for providing information to or receiving input from one or more users of the system 100. In some embodiments, the user interface 170 may include a user input device including, for example, a touch screen, a microphone, a keyboard, a pointing device, a tracking wheel, a camera, knobs, buttons, etc. Using such input devices, a user can provide information input or commands to the system 100 by typing instructions or information, providing voice commands, selecting menu options on a screen using buttons, a pointer, or eye tracking capabilities, or by any other suitable technique for communicating information to the system 100.

[0075] The user interface 170 may be equipped with one or more processing devices configured to provide and receive information to and from a user, and to process the information for use by, for example, the application processor 180. In some embodiments, such processing devices may execute instructions for recognizing and tracking eye movements, receiving and interpreting voice commands, recognizing and interpreting touches and / or gestures made on a touch screen, responding to keyboard input or menu selections, etc. In some embodiments, the user interface 170 may include a display, a speaker, a haptic device, and / or any other device for providing output information to a user.

[0076] The map database 160 may include any type of database for storing map data useful to the system 100. In some embodiments, the map database 160 may include data related to the location of various items in a reference coordinate system, including roads, water features, geographic features, commercial areas, points of interest, restaurants, gas stations, etc. The map database 160 may store not only the location of such items, but also descriptors related to such items, including, for example, names associated with any stored features. In some embodiments, the map database 160 may be physically located with other components of the system 100. Alternatively or additionally, the map database 160 or a portion thereof may be located remotely relative to other components of the system 100 (e.g., the processing unit 110). In such embodiments, information from the map database 160 may be downloaded via a wired or wireless data connection to a network (e.g., via a cellular network and / or the Internet, etc.). In some cases, the map database 160 may store a sparse data model that includes a polynomial representation of certain road features (e.g., lane markings) or a target trajectory of a host vehicle. Systems and methods for generating such maps are discussed below with reference to FIGS. 8 to 19.

[0077] Image capture devices 122, 124, and 126 may each include any type of device suitable for capturing at least one image from an environment. Furthermore, any number of image capture devices may be used to acquire images for input to an image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, or even four or more image capture devices. FIG. 2B to FIG. 2E Image capture devices 122, 124, and 126 are further described.

[0078] The system 100 or its various components may be incorporated into a variety of different platforms. In some embodiments, the system 100 may be included in a vehicle 200, such as Figure 2A For example, vehicle 200 may be equipped with Figure 1 The processing unit 110 and any other components of the system 100 described above. Although in some embodiments, the vehicle 200 may be equipped with only a single image capture device (e.g., a camera), in other embodiments, such as a combination of Figure 2B-2E As discussed above, multiple image capture devices may be used. For example, Figure 2A As shown, either of the image capture devices 122 and 124 of the vehicle 200 may be part of an ADAS (Advanced Driver Assistance System) imaging set.

[0079] The image capture device included on the vehicle 200 as part of the image acquisition unit 120 may be positioned in any suitable location. Figure 2A-2E as well as Figure 3A-3C As shown, the image capture device 122 can be located near the rearview mirror. This location can provide a line of sight similar to that of the driver of the vehicle 200, which can help determine what is visible and invisible to the driver. The image capture device 122 can be positioned in any position near the rearview mirror, and placing the image capture device 122 on the driver's side of the mirror can further help obtain an image representing the driver's field of view and / or line of sight.

[0080] Other positioning can also be used for the image capture device of the image acquisition unit 120. For example, the image capture device 124 can be located on or in the bumper of the vehicle 200. Such positioning can be particularly suitable for image capture devices with a wide field of view. The line of sight of the image capture device located in the bumper can be different from the line of sight of the driver, and therefore, the bumper image capture device and the driver may not always see the same object. The image capture device (e.g., image capture devices 122, 124, and 126) can also be located in other positioning. For example, the image capture device can be located on or in one or both of the side mirrors of the vehicle 200, on the roof of the vehicle 200, on the hood of the vehicle 200, on the trunk of the vehicle 200, on the side of the vehicle 200, installed on any window of the vehicle 200, positioned behind any window of the vehicle 200, or positioned in front of any window of the vehicle 200, and installed in or near the lamps on the front and / or rear of the vehicle 200, etc.

[0081] In addition to the image capture device, the vehicle 200 may also include various other components of the system 100. For example, the processing unit 110 may be included on the vehicle 200, either integrated with or separate from the vehicle's engine control unit (ECU). The vehicle 200 may also be equipped with a location sensor 130, such as a GPS receiver, and may also include a map database 160 and memory units 140 and 150.

[0082] As discussed earlier, the wireless transceiver 172 can transmit and / or receive data over one or more networks (e.g., a cellular network, the Internet, etc.). For example, the wireless transceiver 172 can upload data collected by the system 100 to one or more servers and download data from one or more servers. For example, via the wireless transceiver 172, the system 100 can receive periodic or on-demand updates to data stored in the map database 160, the memory 140, and / or the storage 150. Similarly, the wireless transceiver 172 can upload any data from the system 100 (e.g., images captured by the image acquisition unit 120, data received by the position sensor 130 or other sensors, the vehicle control system, etc.) and / or any data processed by the processing unit 110 to one or more servers.

[0083] The system 100 can upload data to a server (e.g., to a cloud) based on a privacy level setting. For example, the system 100 can implement a privacy level setting to specify or limit the type of data (including metadata) that is transmitted to the server that can uniquely identify the vehicle and / or the driver / owner of the vehicle. Such settings can be set by a user via, for example, the wireless transceiver 172, can be initialized by a factory default setting, or by data received by the wireless transceiver 172.

[0084] In some embodiments, the system 100 may upload data according to a "high" privacy level, and, if set, the system 100 may transmit data (e.g., location information associated with the route, captured images, etc.) without any details about a specific vehicle and / or driver / owner. For example, when uploading data according to a "high" privacy setting, the system 100 may not include the vehicle identification number (VIN) or the name of the driver or owner of the vehicle, and may instead transmit data, such as captured images and / or limited location information associated with the route.

[0085] Other privacy levels are contemplated. For example, the system 100 may transmit data to a server according to an "intermediate" privacy level and may include additional information not included at a "high" privacy level, such as the make and / or model of the vehicle and / or the type of vehicle (e.g., a passenger car, a sport utility vehicle, a truck, etc.). In some embodiments, the system 100 may upload data according to a "low" privacy level. With a "low" privacy level setting, the system 100 may upload data and include information sufficient to uniquely identify a specific vehicle, the owner / driver, and / or a portion or the entire route traveled by the vehicle. Such "low" privacy level data may include one or more of the following: for example, the VIN, the driver / owner's name, the vehicle's point of origin prior to departure, the vehicle's intended destination, the vehicle's make and / or model, the vehicle type, etc.

[0086] Figure 2A is a diagrammatic side view representation of an exemplary vehicle imaging system consistent with the disclosed embodiments. Figure 2B yes Figure 2A A pictorial top view of the embodiment shown in FIG. Figure 2B As shown, the disclosed embodiments may include a vehicle 200 including a system 100 in its body having a first image capture device 122 positioned near a rearview mirror of the vehicle 200 and / or near a driver, a second image capture device 124 positioned on or in a bumper area (e.g., one of the bumper areas 210) of the vehicle 200, and a processing unit 110.

[0087] like Figure 2C As shown, both image capture devices 122 and 124 may be positioned near a rearview mirror of vehicle 200 and / or near the driver. Figure 2B and Figure 2C Two image capture devices 122 and 124 are shown, it should be understood that other embodiments may include more than two image capture devices. Figure 2D and 2E In the illustrated embodiment, a first image capture device 122 , a second image capture device 124 , and a third image capture device 126 are included in the system 100 of a vehicle 200 .

[0088] like Figure 2D As shown, image capture device 122 may be positioned near a rearview mirror of vehicle 200 and / or near the driver, and image capture devices 124 and 126 may be positioned on or in a bumper region (e.g., one of bumper regions 210) of vehicle 200. Figure 2E As shown, image capture devices 122, 124, and 126 may be positioned near rearview mirrors and / or near the driver's seat of vehicle 200. The disclosed embodiments are not limited to any particular number and configuration of image capture devices, and the image capture devices may be positioned in any suitable location within or on vehicle 200.

[0089] It should be understood that the disclosed embodiments are not limited to vehicles and can be applied in other scenarios. It should also be understood that the disclosed embodiments are not limited to a specific type of vehicle 200 and can be applicable to all types of vehicles, including cars, trucks, trailers and other types of vehicles.

[0090] The first image capture device 122 may include any suitable type of image capture device. The image capture device 122 may include an optical axis. In one example, the image capture device 122 may include an Aptina M9V024WVGA sensor with a global shutter. In other embodiments, the image capture device 122 may provide a resolution of 1280×960 pixels and may include a rolling shutter. The image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included, for example, to provide the image capture device with a desired focal length and field of view. In some embodiments, the image capture device 122 may be associated with a 6 mm lens or a 12 mm lens. In some embodiments, as Figure 2DAs shown, the image capture device 122 can be configured to capture an image with a desired field of view (FOV) 202. For example, the image capture device 122 can be configured to have a conventional FOV, such as in the range of 40 degrees to 56 degrees, including a 46 degree FOV, a 50 degree FOV, a 52 degree FOV, or a larger FOV. Alternatively, the image capture device 122 can be configured to have a narrow FOV in the range of 23 degrees to 40 degrees, such as a 28 degree FOV or a 36 degree FOV. In addition, the image capture device 122 can be configured to have a wide FOV in the range of 100 degrees to 180 degrees. In some embodiments, the image capture device 122 may include a wide-angle bumper camera or a camera with up to 180 degrees FOV. In some embodiments, the image capture device 122 may be a 7.2M pixel image capture device with an aspect ratio of approximately 2:1 (e.g., HxV=3800×1900 pixels) with a horizontal FOV of approximately 100 degrees. Such an image capture device may be used in place of a three image capture device configuration. Due to significant lens distortion, in embodiments where the image capture device uses a radially symmetric lens, the vertical FOV of such an image capture device may be significantly less than 50 degrees. For example, such a lens may not be radially symmetric, which would allow a vertical FOV greater than 50 degrees with a 100 degree horizontal FOV.

[0091] The first image capture device 122 may acquire a plurality of first images of a scene associated with the vehicle 200. Each of the plurality of first images may be acquired as a series of image scan lines, which may be captured using a rolling shutter. Each scan line may contain a plurality of pixels.

[0092] The first image capture device 122 may have a scan rate associated with the acquisition of each of the first series of image scan lines. The scan rate may refer to the rate at which the image sensor may acquire image data associated with each pixel contained in a particular scan line.

[0093] Image capture devices 122, 124, and 126 may contain any suitable type and number of image sensors, including, for example, CCD sensors or CMOS sensors, etc. In one embodiment, a CMOS image sensor may be employed with a rolling shutter so that each pixel in a row is read one at a time, and the scanning of the rows is performed on a row-by-row basis until the entire image frame has been captured. In some embodiments, the rows may be captured sequentially from top to bottom with respect to the frame.

[0094] In some embodiments, one or more of the image capture devices disclosed herein (e.g., image capture devices 122, 124, and 126) may constitute a high-resolution imager and may have a resolution greater than 5M pixels, 7M pixels, 10M pixels, or more.

[0095] The use of a rolling shutter may cause pixels in different rows to be exposed and captured at different times, which may cause skew and other image artifacts in the captured image frame. On the other hand, when the image capture device 122 is configured to operate with a global or synchronized shutter, all pixels may be exposed for the same amount of time and during a common exposure period. As a result, the image data in a frame collected from a system employing a global shutter represents a snapshot of the entire FOV (such as FOV 202) at a particular time. In contrast, in a rolling shutter application, each row in the frame is exposed and data is captured at a different time. As a result, moving objects may appear distorted in an image capture device with a rolling shutter. This phenomenon will be described in more detail below.

[0096] The second image capture device 124 and the third image capture device 126 may be any type of image capture device. Similar to the first image capture device 122, each of the image capture devices 124 and 126 may include an optical axis. In one embodiment, each of the image capture devices 124 and 126 may include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of the image capture devices 124 and 126 may include a rolling shutter. Similar to the image capture device 122, the image capture devices 124 and 126 may be configured to include various lenses and optical elements. In some embodiments, the lenses associated with the image capture devices 124 and 126 may provide a FOV (such as FOV 204 and 206) that is equal to or narrower than the FOV (such as FOV 202) associated with the image capture device 122. For example, the image capture devices 124 and 126 may have a FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less.

[0097] Image capture devices 124 and 126 may acquire a plurality of second and third images of a scene associated with vehicle 200. Each of the plurality of second and third images may be acquired as a second and third series of image scan lines, which may be captured using a rolling shutter. Each scan line or row may have a plurality of pixels. Image capture devices 124 and 126 may have second and third scan rates associated with the acquisition of each image scan line included in the second and third series.

[0098] Each image capture device 122, 124, and 126 may be positioned at any suitable location and orientation relative to vehicle 200. The relative locations of image capture devices 122, 124, and 126 may be selected to help fuse together information obtained from the image capture devices. For example, in some embodiments, a FOV associated with image capture device 124 (such as FOV 204) may partially or completely overlap with a FOV associated with image capture device 122 (e.g., FOV 202) and a FOV associated with image capture device 126 (e.g., FOV 206).

[0099] Image capture devices 122, 124, and 126 may be located at any suitable relative heights on vehicle 200. In one example, there may be height differences between image capture devices 122, 124, and 126 that may provide sufficient parallax information to enable stereo analysis. Figure 2A As shown, the two image capture devices 122 and 124 are at different heights. For example, there may also be a lateral displacement difference between the image capture devices 122, 124, and 126 to provide additional parallax information for the stereo analysis of the processing unit 110. The difference in lateral displacement can be represented by dx, as Figure 2C and Figure 2D In some embodiments, there may be a forward or backward displacement (e.g., a range displacement) between image capture devices 122, 124, and 126. For example, image capture device 122 may be located 0.5 to 2 meters or more behind image capture device 124 and / or image capture device 126. This type of displacement may enable one of the image capture devices to cover a potential blind spot of the other (one or more) image capture devices.

[0100] Image capture device 122 may have any suitable resolution capability (e.g., the number of pixels associated with the image sensor), and the resolution of the image sensor(s) associated with image capture device 122 may be higher, lower, or the same as the resolution of the image sensor(s) associated with image capture devices 124 and 126. In some embodiments, the image sensor(s) associated with image capture device 122 and / or image capture devices 124 and 126 may have a resolution of 640×480, 1024×768, 1280×960, or any other suitable resolution.

[0101] The frame rate (e.g., the rate at which an image capture device acquires a set of pixel data for one image frame before continuing to capture pixel data associated with the next image frame) may be controllable. The frame rate associated with image capture device 122 may be higher, lower, or the same as the frame rate associated with image capture devices 124 and 126. The frame rate associated with image capture devices 122, 124, and 126 may depend on various factors that may affect the timing of the frame rate. For example, one or more of image capture devices 122, 124, and 126 may include a selectable pixel delay period that is applied before or after acquiring image data associated with one or more pixels of an image sensor in image capture devices 122, 124, and / or 126. Typically, image data corresponding to each pixel may be acquired based on a clock rate for the device (e.g., one pixel per clock cycle). Furthermore, in embodiments that include a rolling shutter, one or more of the image capture devices 122, 124, and 126 may include a selectable horizontal blanking period that is applied before or after acquiring image data associated with a row of pixels of an image sensor in the image capture devices 122, 124, and / or 126. Furthermore, one or more of the image capture devices 122, 124, and / or 126 may include a selectable vertical blanking period that is applied before or after acquiring image data associated with an image frame of the image capture devices 122, 124, and 126.

[0102] These timing controls may enable synchronization of frame rates associated with image capture devices 122, 124, and 126, even if the line scan rates of each are different. Furthermore, as will be discussed in more detail below, these selectable timing controls, as well as other factors (e.g., image sensor resolution, maximum line scan rate, etc.), may enable synchronization of image capture from areas where the FOV of image capture device 122 overlaps with one or more FOVs of image capture devices 124 and 126, even if the field of view of image capture device 122 is different than the FOVs of image capture devices 124 and 126.

[0103] The frame rate timing in image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensors. For example, assuming similar line scan rates for both devices, if one device includes an image sensor with a resolution of 640x480 and the other device includes an image sensor with a resolution of 1280x960, more time will be required to acquire a frame of image data from the sensor with the higher resolution.

[0104] Another factor that can affect the timing of image data acquisition in image capture devices 122, 124, and 126 is the maximum line scan rate. For example, it will take a certain minimum amount of time to acquire a line of image data from the image sensors included in image capture devices 122, 124, and 126. Assuming that no pixel delay period is added, this minimum amount of time for acquiring a line of image data will be related to the maximum line scan rate for the particular device. Devices that provide higher maximum line scan rates have the potential to provide higher frame rates than devices with lower maximum line scan rates. In some embodiments, one or more of image capture devices 124 and 126 may have a maximum line scan rate that is higher than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of image capture devices 124 and / or 126 may be 1.25, 1.5, 1.75, or 2 times or more the maximum line scan rate of image capture device 122.

[0105] In another embodiment, image capture devices 122, 124, and 126 may have the same maximum line scan rate, but image capture device 122 may operate at a scan rate that is less than or equal to its maximum scan scan rate. The system may be configured such that one or more of image capture devices 124 and 126 operate at a line scan rate that is equal to the line scan rate of image capture device 122. In other examples, the system may be configured such that the line scan rate of image capture device 124 and / or image capture device 126 may be 1.25, 1.5, 1.75, or 2 or more times the line scan rate of image capture device 122.

[0106] In some embodiments, image capture devices 122, 124, and 126 may be asymmetric. In other words, they may include cameras with different fields of view (FOV) and focal lengths. For example, the fields of view of image capture devices 122, 124, and 126 may include any desired area of ​​the environment of vehicle 200. In some embodiments, one or more of image capture devices 122, 124, and 126 may be configured to acquire image data from the environment in front of vehicle 200, behind vehicle 200, to the side of vehicle 200, or a combination thereof.

[0107] In addition, the focal length associated with each image capture device 122, 124, and / or 126 may be selectable (e.g., by including an appropriate lens, etc.) so that each device acquires images of objects at a desired range of distances relative to the vehicle 200. For example, in some embodiments, the image capture devices 122, 124, and 126 may acquire images of objects that are within a few meters of the vehicle. The image capture devices 122, 124, and 126 may also be configured to acquire images of objects at a greater range from the vehicle (e.g., 25 meters, 50 meters, 100 meters, 150 meters, or more). In addition, the focal lengths of image capture devices 122, 124, and 126 may be selected so that one image capture device (e.g., image capture device 122) may capture images of objects that are relatively close to the vehicle (e.g., within 10 meters or within 20 meters), while other image capture devices (e.g., image capture devices 124 and 126) may capture images of objects that are farther away from the vehicle 200 (e.g., greater than 20 meters, 50 meters, 100 meters, 150 meters, etc.).

[0108] According to some embodiments, the FOV of one or more image capture devices 122, 124, and 126 may have a wide angle. For example, having a FOV of 140 degrees may be advantageous, especially for image capture devices 122, 124, and 126 that may be used to capture images of areas near vehicle 200. For example, image capture device 122 may be used to capture images of areas to the right or left of vehicle 200, and in such embodiments, it may be desirable for image capture device 122 to have a wide FOV (e.g., at least 140 degrees).

[0109] The field of view associated with each of the image capture devices 122, 124, and 126 may depend on the corresponding focal length. For example, as the focal length increases, the corresponding field of view decreases.

[0110] Image capture devices 122, 124, and 126 may be configured to have any suitable field of view. In one particular example, image capture device 122 may have a horizontal FOV of 46 degrees, image capture device 124 may have a horizontal FOV of 23 degrees, and image capture device 126 may have a horizontal FOV between 23 degrees and 46 degrees. In one particular example, image capture device 122 may have a horizontal FOV of 52 degrees, image capture device 124 may have a horizontal FOV of 26 degrees, and image capture device 126 may have a horizontal FOV between 26 degrees and 52 degrees. In some embodiments, the ratio of the FOV of image capture device 122 to the FOV of image capture device 124 and / or image capture device 126 may vary from 1.5 to 2.0. In other embodiments, the ratio may vary between 1.25 and 2.25.

[0111] The system 100 may be configured such that the field of view of the image capture device 122 at least partially or completely overlaps the field of view of the image capture device 124 and / or the image capture device 126. In some embodiments, the system 100 may be configured such that the fields of view of the image capture devices 124 and 126, for example, fall within (e.g., are narrower than) the field of view of the image capture device 122 and share a common center with the field of view of the image capture device 122. In other embodiments, the image capture devices 122, 124, and 126 may capture adjacent FOVs, or may have partial overlap in their FOVs. In some embodiments, the fields of view of the image capture devices 122, 124, and 126 may be aligned such that the center of the narrower FOV image capture device 124 and / or 126 may be located in the lower half of the field of view of the wider FOV device 122.

[0112] Figure 2F is a pictorial representation of an exemplary vehicle control system consistent with the disclosed embodiments. Figure 2F As indicated, the vehicle 200 may include a throttle adjustment system 220, a braking system 230, and a steering system 240. The system 100 may provide input (e.g., control signals) to one or more of the throttle adjustment system 220, the braking system 230, and the steering system 240 via one or more data links (e.g., any wired and / or wireless links for transmitting data). For example, based on an analysis of images acquired by the image capture devices 122, 124, and / or 126, the system 100 may provide control signals to one or more of the throttle adjustment system 220, the braking system 230, and the steering system 240 to navigate the vehicle 200 (e.g., by causing acceleration, steering, lane shifting, etc.). In addition, the system 100 may receive input from one or more of the throttle adjustment system 220, the braking system 230, and the steering system 240 that indicates an operating condition of the vehicle 200 (e.g., speed, whether the vehicle 200 is braking and / or turning, etc.). The following is combined with Figures 4 to 7 Provide further details.

[0113] like Figure 3AAs shown, the vehicle 200 may also include a user interface 170 for interacting with a driver or passenger of the vehicle 200. For example, the user interface 170 in the vehicle application may include a touch screen 320, a knob 330, a button 340, and a microphone 350. The driver or passenger of the vehicle 200 may also interact with the system 100 using a handle (e.g., located on or near the steering column of the vehicle 200, including, for example, a turn signal handle), a button (e.g., located on the steering wheel of the vehicle 200), etc. In some embodiments, the microphone 350 may be positioned adjacent to the rearview mirror 310. Similarly, in some embodiments, the image capture device 122 may be located near the rearview mirror 310. In some embodiments, the user interface 170 may also include one or more speakers 360 (e.g., speakers of a vehicle audio system). For example, the system 100 may provide various notifications (e.g., alarms) via the speaker 360.

[0114] FIG. 3B to FIG. 3D is an illustration of an exemplary camera mount 370 configured to be positioned behind a rearview mirror (e.g., rearview mirror 310) and against a vehicle windshield consistent with the disclosed embodiments. Figure 3B As shown, camera mount 370 can include image capture devices 122, 124, and 126. Image capture devices 124 and 126 can be positioned behind visor 380, which can be flush with the vehicle windshield and include a film and / or a composite of anti-reflective materials. For example, visor 380 can be positioned so that the shield is aligned relative to the vehicle windshield with a matching slope. In some embodiments, each of image capture devices 122, 124, and 126 can be positioned behind visor 380, such as at Figure 3D The disclosed embodiments are not limited to any particular configuration of image capture devices 122 , 124 , and 126 , camera mount 370 , and visor 380 . Figure 3C yes Figure 3B The camera mount 370 is shown as an illustration from a front perspective.

[0115] As will be appreciated by those skilled in the art having the benefit of this disclosure, many changes and / or modifications may be made to the foregoing disclosed embodiments. For example, not all components are necessary for the operation of the system 100. In addition, any component may be located in any appropriate portion of the system 100 and the components may be rearranged into various configurations while providing the functionality of the disclosed embodiments. Therefore, the foregoing configurations are exemplary, and regardless of the configurations discussed above, the system 100 may provide a wide range of functionality to analyze the surroundings of the vehicle 200 and navigate the vehicle 200 in response to the analysis.

[0116] As discussed in more detail below and in accordance with various disclosed embodiments, the system 100 can provide various features related to autonomous driving and / or driver assistance technology. For example, the system 100 can analyze image data, location data (e.g., GPS location information), map data, speed data, and / or data from sensors included in the vehicle 200. The system 100 can collect data for analysis from, for example, the image acquisition unit 120, the location sensor 130, and other sensors. In addition, the system 100 can analyze the collected data to determine whether the vehicle 200 should take a certain action, and then automatically take the determined action without human intervention. For example, when the vehicle 200 is navigating without human intervention, the system 100 can automatically control the braking, acceleration, and / or steering of the vehicle 200 (e.g., by transmitting control signals to one or more of the throttle adjustment system 220, the braking system 230, and the steering system 240). In addition, the system 100 can analyze the collected data and issue warnings and / or alarms to the vehicle occupants based on the analysis of the collected data. Additional details of various embodiments provided by the system 100 are provided below.

[0117] Forward multi-imaging system

[0118] As discussed above, the system 100 can provide a driving assistance function using a multi-camera system. The multi-camera system can use one or more cameras facing the front of the vehicle. In other embodiments, the multi-camera system can include one or more cameras facing the side of the vehicle or facing the rear of the vehicle. In one embodiment, for example, the system 100 can use a dual-camera imaging system, wherein the first camera and the second camera (e.g., image capture devices 122 and 124) can be positioned in front of and / or on the side of the vehicle (e.g., vehicle 200). The first camera can have a field of view that is larger than, smaller than, or partially overlaps the field of view of the second camera. In addition, the first camera can be connected to a first image processor to perform a monocular image analysis of the image provided by the first camera, and the second camera can be connected to a second image processor to perform a monocular image analysis of the image provided by the second camera. The outputs of the first and second image processors (e.g., processed information) can be combined. In some embodiments, the second image processor can receive images from both the first camera and the second camera to perform stereo analysis. In another embodiment, the system 100 can use a three-camera imaging system, wherein each camera has a different field of view. Thus, such a system may make decisions based on information derived from objects located at different distances in front of and to the side of the vehicle. References to monocular image analysis may refer to instances of image analysis based on images captured from a single viewpoint (e.g., from a single camera). Stereoscopic image analysis may refer to instances of image analysis based on two or more images captured using one or more changes in image capture parameters. For example, captured images suitable for stereoscopic image analysis may include images captured from two or more different positions, from different fields of view, using different focal lengths, and together with parallax information, etc.

[0119] For example, in one embodiment, the system 100 may implement a three-camera configuration using image capture devices 122, 124, and 126. In such a configuration, image capture device 122 may provide a narrow field of view (e.g., 34 degrees or other values ​​selected from a range of about 20 degrees to 45 degrees, etc.), image capture device 124 may provide a wide field of view (e.g., 150 degrees or other values ​​selected from a range of about 100 degrees to about 180 degrees), and image capture device 126 may provide an intermediate field of view (e.g., 46 degrees or other values ​​selected from a range of about 35 degrees to about 60 degrees). In some embodiments, image capture device 126 may serve as a primary camera or base camera. Image capture devices 122, 124, and 126 may be positioned behind rearview mirror 310 and positioned substantially side by side (e.g., 6 centimeters apart). In addition, in some embodiments, as discussed above, one or more of image capture devices 122, 124, and 126 may be mounted behind a sun visor 380 flush with the windshield of vehicle 200. Such shielding may be used to minimize the effect of any reflections from the interior of the vehicle on image capture devices 122 , 124 , and 126 .

[0120] In another embodiment, as above combined Figure 3B and Figure 3C As discussed, the wide field of view camera (e.g., image capture device 124 in the above example) can be mounted lower than the narrow field of view camera and the main field of view camera (e.g., image capture devices 122 and 126 in the above example). Such a configuration can provide a free line of sight from the wide field of view camera. To reduce reflections, the camera can be mounted closer to the windshield of the vehicle 200, and a polarizer can be included on the camera to dampen reflected light.

[0121] The three-camera system may provide certain performance characteristics. For example, some embodiments may include the ability to verify the detection of an object by one camera based on the detection results from another camera. In the three-camera configuration discussed above, the processing unit 110 may include, for example, three processing devices (e.g., three EyeQ series processor chips as discussed above), where each processing device is dedicated to processing images captured by one or more of the image capture devices 122, 124, and 126.

[0122] In a three-camera system, the first processing device can receive images from both the main camera and the narrow field of view camera and perform visual processing for the narrow FOV camera, for example to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. In addition, the first processing device can calculate the disparity of pixels between the images from the main camera and the narrow camera and create a 3D reconstruction of the environment of the vehicle 200. The first processing device can then combine the 3D reconstruction with 3D map data, or combine the 3D reconstruction with 3D information calculated based on information from another camera.

[0123] The second processing device can receive images from the main camera and perform visual processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. In addition, the second processing device can calculate the camera displacement, and based on the displacement, calculate the disparity of pixels between consecutive images and create a 3D reconstruction of the scene (e.g., structure from motion). The second processing device can transmit the structure from motion based on the 3D reconstruction to the first processing device to be combined with the stereoscopic 3D image.

[0124] The third processing device can receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other road objects. The third processing device can further execute additional processing instructions to analyze the images to identify moving objects in the images, such as vehicles changing lanes, pedestrians, etc.

[0125] In some embodiments, having the streams of image-based information captured and processed independently can provide an opportunity to provide redundancy in the system. Such redundancy can include, for example, using a first image capture device and images processed from that device to verify and / or supplement information obtained by capturing and processing image information from at least a second image capture device.

[0126] In some embodiments, system 100 may use two image capture devices (e.g., image capture devices 122 and 124) to provide navigation assistance for vehicle 200, and use a third image capture device (e.g., image capture device 126) to provide redundancy and verify the analysis of data received from the other two image capture devices. For example, in such a configuration, image capture devices 122 and 124 may provide images for stereo analysis by system 100 for navigating vehicle 200, while image capture device 126 may provide images for monocular analysis by system 100 to provide redundancy and verification of information obtained based on images captured from image capture device 122 and / or image capture device 124. In other words, image capture device 126 (and corresponding processing device) may be considered to provide a redundant subsystem for providing a check on the analysis derived from image capture devices 122 and 124 (e.g., to provide an automatic emergency braking (AEB) system). Additionally, in some embodiments, redundancy and verification of received data may be supplemented based on information received from one or more sensors (e.g., radar, lidar, acoustic sensors, information received from one or more transceivers outside the vehicle, etc.).

[0127] Those skilled in the art will recognize that the above-described camera configurations, camera placements, number of cameras, camera positioning, etc. are merely examples. These components and other components described with respect to the overall system may be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of a multi-camera system to provide driver assistance and / or autonomous vehicle functionality are as follows.

[0128] Figure 4 1 is an exemplary functional block diagram of memory 140 and / or memory 150, which may be stored / programmed with instructions for performing one or more operations consistent with the embodiments of the present disclosure. Although the following refers to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or memory 150.

[0129] like Figure 4As shown, the memory 140 may store a monocular image analysis module 402, a stereo image analysis module 404, a velocity and acceleration module 406, and a navigation response module 408. The disclosed embodiments are not limited to any particular configuration of the memory 140. In addition, the application processor 180 and / or the image processor 190 may execute instructions stored in any of the modules 402, 404, 406, and 408 contained in the memory 140. Those skilled in the art will appreciate that references to the processing unit 110 in the following discussion may refer to the application processor 180 and the image processor 190 separately or collectively. Accordingly, the steps of any of the following processes may be performed by one or more processing devices.

[0130] In one embodiment, monocular image analysis module 402 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform monocular image analysis on a set of images acquired by one of image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from a set of images with additional sensory information (e.g., information from radar, lidar, etc.) to perform monocular image analysis. FIG. 5A to FIG. 5D As described, the monocular image analysis module 402 may include instructions for detecting a set of features within a set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other features associated with the vehicle's environment. Based on the analysis, the system 100 (e.g., via the processing unit 110) may cause one or more navigation responses in the vehicle 200, such as steering, lane changes, changes in acceleration, etc., as discussed below in conjunction with the navigation response module 408.

[0131] In one embodiment, stereo image analysis module 404 may store instructions (such as computer vision software) that, when executed by processing unit 110, perform stereo image analysis on a first set of second sets of images acquired by a combination of image capture devices selected from any of image capture devices 122, 124, and 126. In some embodiments, processing unit 110 may combine information from the first set of second sets of images with additional sensory information (e.g., information from radar) to perform stereo image analysis. For example, stereo image analysis module 404 may include instructions for performing stereo image analysis based on a first set of images acquired by image capture device 124 and a second set of images acquired by image capture device 126. As described below in conjunction with Figure 6As described, the stereo image analysis module 404 may include instructions for detecting a set of features within the first and second sets of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, etc. Based on the analysis, the processing unit 110 may cause one or more navigation responses in the vehicle 200, such as steering, changing lanes, changes in acceleration, etc., as discussed below in conjunction with the navigation response module 408. In addition, in some embodiments, the stereo image analysis module 404 may implement techniques associated with a trained system (such as a neural network or a deep neural network) or an untrained system, such as a system that may be configured to use a computer vision algorithm to detect and / or mark objects in an environment from which sensory information is captured and processed. In one embodiment, the stereo image analysis module 404 and / or other image processing modules may be configured to use a combination of trained and untrained systems.

[0132] In one embodiment, the speed and acceleration module 406 may store software configured to analyze data received from one or more computing and electromechanical devices in the vehicle 200 that are configured to cause changes in the speed and / or acceleration of the vehicle 200. For example, the processing unit 110 may execute instructions associated with the speed and acceleration module 406 to calculate a target speed for the vehicle 200 based on data derived from executing the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include, for example, target position, speed, and / or acceleration, the position and / or speed of the vehicle 200 relative to nearby vehicles, pedestrians, or road objects, position information of the vehicle 200 relative to lane markings of the road, etc. In addition, the processing unit 110 may calculate the target speed for the vehicle 200 based on sensory input (e.g., information from a radar) and input from other systems of the vehicle 200, such as the throttle adjustment system 220, the braking system 230, and / or the steering system 240. Based on the calculated target speed, the processing unit 110 can transmit an electrical signal to the throttle regulation system 220, the braking system 230 and / or the steering system 240 of the vehicle 200 to trigger a change in speed and / or acceleration by, for example, physically pressing the brake or releasing the accelerator of the vehicle 200.

[0133] In one embodiment, the navigation response module 408 may store software that is executable by the processing unit 110 to determine a desired navigation response based on data derived from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. Such data may include position and velocity information associated with nearby vehicles, pedestrians, and road objects, target position information for the vehicle 200, etc. In addition, in some embodiments, the navigation response may be based (partially or entirely) on map data, a predetermined position of the vehicle 200, and / or a relative velocity or relative acceleration between the vehicle 200 and one or more objects detected from the execution of the monocular image analysis module 402 and / or the stereo image analysis module 404. The navigation response module 408 may also determine the desired navigation response based on sensory input (e.g., information from a radar) and input from other systems of the vehicle 200, such as the throttle adjustment system 220, the braking system 230, and the steering system 240 of the vehicle 200. Based on the desired navigation response, the processing unit 110 may transmit electrical signals to the throttle adjustment system 220, the braking system 230, and the steering system 240 of the vehicle 200 to achieve a predetermined angle of rotation by, for example, turning the steering wheel of the vehicle 200, thereby triggering the desired navigation response. In some embodiments, the processing unit 110 may use the output of the navigation response module 408 (e.g., the desired navigation response) as an input to execute the speed and acceleration module 406 for calculating the change in the speed of the vehicle 200.

[0134] Furthermore, any modules disclosed herein (eg, modules 402, 404, and 406) may implement techniques associated with a trained system (such as a neural network or a deep neural network) or an untrained system.

[0135] Figure 5A is a flow chart illustrating an exemplary process 500A for causing one or more navigation responses based on monocular image analysis consistent with the disclosed embodiments. At step 510, the processing unit 110 may receive a plurality of images via a data interface 128 between the processing unit 110 and the image acquisition unit 120. For example, a camera (such as an image capture device 122 having a field of view 202) included in the image acquisition unit 120 may capture a plurality of images of an area in front of the vehicle 200 (e.g., or to the side or rear of the vehicle) and transmit them to the processing unit 110 via a data connection (e.g., digital, wired, USB, wireless, Bluetooth, etc.). At step 520, the processing unit 110 may execute the monocular image analysis module 402 to analyze the plurality of images, as described below in conjunction with the processing unit 110. FIG. 5B to FIG. 5D By performing this analysis, processing unit 110 can detect a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, etc.

[0136] At step 520, processing unit 110 may also execute monocular image analysis module 402 to detect various road hazards, such as, for example, parts of truck tires, fallen road signs, loose cargo, small animals, etc. Road hazards may vary in structure, shape, size, and color, which may make the detection of such hazards more difficult. In some embodiments, processing unit 110 may execute monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, processing unit 110 may estimate camera motion between consecutive image frames and calculate the parallax in pixels between frames to build a 3D map of the road. Processing unit 110 may then use the 3D map to detect the road surface and the hazards present on the road surface.

[0137] At step 530, the processing unit 110 may execute the navigation response module 408 to perform the navigation response based on the analysis performed at step 520 and the above combination. Figure 4 The described techniques are used to cause one or more navigation responses. The navigation response may include, for example, steering, changing lanes, braking, changes in acceleration, etc. In some embodiments, the processing unit 110 may use data derived from the execution speed and acceleration module 406 to cause the one or more navigation responses. In addition, multiple navigation responses may occur simultaneously, sequentially, or in any combination thereof. For example, the processing unit 110 may transmit control signals to the steering system 240 and the throttle adjustment system 220 of the vehicle 200, for example, in sequence, so that the vehicle 200 changes a lane and then accelerates. Alternatively, the processing unit 110 may transmit control signals to the braking system 230 and the steering system 240 of the vehicle 200, for example, at the same time, so that the vehicle 200 brakes while changing lanes.

[0138] Figure 5B is a flow chart showing an exemplary process 500B for detecting one or more vehicles and / or pedestrians in a set of images consistent with the disclosed embodiments. The processing unit 110 may execute the monocular image analysis module 402 to implement the process 500B. At step 540, the processing unit 110 may determine a set of candidate objects representing possible vehicles and / or pedestrians. For example, the processing unit 110 may scan one or more images, compare the images to one or more predetermined patterns, and identify possible locations within each image that may contain objects of interest (e.g., vehicles, pedestrians, or parts thereof). The predetermined patterns may be designed in such a way as to achieve a high "false hit" rate and a low "miss" rate. For example, the processing unit 110 may use a low similarity threshold on the predetermined pattern to identify the candidate object as a possible vehicle or pedestrian. Doing so may allow the processing unit 110 to reduce the likelihood of missing (e.g., not identifying) a candidate object representing a vehicle or pedestrian.

[0139] At step 542, processing unit 110 may filter the set of candidate objects based on classification criteria to exclude certain candidates (e.g., irrelevant or less relevant objects). Such criteria may be derived from various attributes associated with object type categories stored in a database (e.g., a database stored in memory 140). Attributes may include object shape, size, texture, location (e.g., relative to vehicle 200), etc. Thus, processing unit 110 may use one or more sets of criteria to reject pseudo candidates from the set of candidate objects.

[0140] At step 544, processing unit 110 may analyze multiple frames of images to determine whether an object in the set of candidate objects represents a vehicle and / or a pedestrian. For example, processing unit 110 may track detected candidate objects across consecutive frames and accumulate frame-by-frame data associated with the detected objects (e.g., size, position relative to vehicle 200, etc.). In addition, processing unit 110 may estimate parameters of the detected object and compare the frame-by-frame position data of the object with the predicted position.

[0141] At step 546, processing unit 110 may construct a set of measurements for the detected objects. Such measurements may include, for example, position, velocity, and acceleration values ​​associated with the detected objects (relative to vehicle 200). In some embodiments, processing unit 110 may construct the measurements based on estimation techniques such as a Kalman filter or linear quadratic estimation (LQE) using a series of time-based observations and / or based on modeling data available for different object categories (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). The Kalman filter may be based on a measure of the scale of the object, where the scale measure is proportional to the time to collision (e.g., the amount of time it takes vehicle 200 to reach the object). Thus, by performing steps 540 to 546, processing unit 110 may identify vehicles and pedestrians that appear within the set of captured images and derive information associated with the vehicles and pedestrians (e.g., position, velocity, size). Based on the identification and the derived information, processing unit 110 may cause one or more navigation responses in vehicle 200, such as those described above in conjunction with Figure 5A as described.

[0142] At step 548, the processing unit 110 may perform an optical flow analysis of the one or more images to reduce the likelihood of detecting "false hits" and missing candidate objects representing vehicles or pedestrians. Optical flow analysis may refer to, for example, analyzing motion patterns relative to the vehicle 200, associated with other vehicles and pedestrians, and distinct from the motion of the road surface in one or more images. The processing unit 110 may calculate the motion of the candidate object by observing different positions of the object across multiple image frames captured at different times. The processing unit 110 may use the position and time values ​​as input to a mathematical model for calculating the motion of the candidate object. Therefore, optical flow analysis may provide another method for detecting vehicles and pedestrians near the vehicle 200. The processing unit 110 may perform optical flow analysis in conjunction with steps 540 to 546 to provide redundancy for detecting vehicles and pedestrians and improve the reliability of the system 100.

[0143] Figure 5C is a flow chart illustrating an exemplary process 500C for detecting road signs and / or lane geometry information in a set of images consistent with the disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500C. At step 550, processing unit 110 may detect a set of objects by scanning one or more images. In order to detect lane signs, lane geometry information, and other relevant road sign segments, processing unit 110 may filter the set of objects to exclude objects determined to be irrelevant (e.g., small potholes, small rocks, etc.). At step 552, processing unit 110 may group together the road segments that belong to the same road sign or lane sign detected in step 550. Based on the grouping, processing unit 110 may generate a model, such as a mathematical model, representing the detected road segments.

[0144] At step 554, the processing unit 110 may construct a set of measurements associated with the detected road segment. In some embodiments, the processing unit 110 may create a projection of the detected road segment from the image plane onto the real world plane. The projection may be characterized using a cubic polynomial with coefficients corresponding to physical properties such as the position, slope, curvature, and curvature derivative of the detected road. When generating the projection, the processing unit 110 may take into account changes in the road surface, as well as the pitch and roll rate associated with the vehicle 200. In addition, the processing unit 110 may model the road elevation by analyzing the position and motion cues present on the road surface. In addition, the processing unit 110 may estimate the pitch and roll rate associated with the vehicle 200 by tracking a set of feature points in one or more images.

[0145] At step 556, processing unit 110 may perform a multi-frame analysis by, for example, tracking the detected road segments across consecutive image frames and accumulating frame-by-frame data associated with the detected road segments. As processing unit 110 performs a multi-frame analysis, the set of measurements constructed at step 554 may become more reliable and associated with increasingly higher confidence levels. Thus, by performing steps 550, 552, 554, and 556, processing unit 110 may identify road signs that appear in the set of captured images and derive lane geometry information. Based on the identification and derived information, processing unit 110 may cause one or more navigation responses in vehicle 200, such as those described above in conjunction with Figure 5A as described.

[0146] At step 558, processing unit 110 may consider additional information sources to further generate a safety model of vehicle 200 in its surrounding environment. Processing unit 110 may use the safety model to define a context in which system 100 may perform autonomous control of vehicle 200 in a safe manner. To generate the safety model, in some embodiments, processing unit 110 may consider the positions and motions of other vehicles, detected curbs and guardrails, and / or general road shape descriptions extracted from map data (such as data from map database 160). By considering additional information sources, processing unit 110 may provide redundancy for detecting road signs and lane geometry and increase the reliability of system 100.

[0147] Figure 5D 5 is a flow chart showing an exemplary process 500D for detecting a traffic light in a set of images consistent with the disclosed embodiments. The processing unit 110 may execute the monocular image analysis module 402 to implement the process 500D. At step 560, the processing unit 110 may scan the set of images and identify objects that appear in the image at locations that may contain traffic lights. For example, the processing unit 110 may filter the identified objects to construct a set of candidate objects, while excluding those objects that are unlikely to correspond to traffic lights. The filtering may be done based on various attributes associated with traffic lights, such as shape, size, texture, position (e.g., relative to the vehicle 200), etc. Such attributes may be based on multiple examples of traffic lights and traffic control signals and stored in a database. In some embodiments, the processing unit 110 may perform a multi-frame analysis on the set of candidate objects reflecting possible traffic lights. For example, the processing unit 110 may track candidate objects across consecutive image frames, estimate the real-world position of the candidate objects, and filter out those moving objects (which are unlikely to be traffic lights). In some embodiments, processing unit 110 may perform color analysis on the candidate objects and identify relative locations of detected colors that appear within possible traffic lights.

[0148] At step 562, processing unit 110 may analyze the geometry of the intersection. The analysis may be based on any combination of (i) the number of lanes detected on either side of vehicle 200, (ii) signs detected on the road (e.g., arrow signs), and (iii) a description of the intersection extracted from map data (e.g., data from map database 160). Processing unit 110 may use information derived from performing monocular analysis module 402 to perform the analysis. In addition, processing unit 110 may determine the correspondence between the traffic lights detected at step 560 and the lanes present near vehicle 200.

[0149] At step 564, as vehicle 200 approaches the intersection, processing unit 110 may update a confidence level associated with the analyzed intersection geometry and detected traffic lights. For example, the number of traffic lights estimated to be present at the intersection compared to the number of traffic lights actually present at the intersection may affect the confidence level. Therefore, based on the confidence level, processing unit 110 may delegate control to the driver of vehicle 200 in order to improve safety conditions. By performing steps 560, 562, and 564, processing unit 110 may identify traffic lights that appear within the set of captured images and analyze intersection geometry information. Based on this identification and analysis, processing unit 110 may cause one or more navigation responses in vehicle 200, such as those described above in conjunction with Figure 5A as described.

[0150] Figure 5E 5 is a flow chart showing an exemplary process 500E for causing one or more navigation responses in vehicle 200 based on a vehicle path consistent with the disclosed embodiments. At step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. The vehicle path may be represented by a set of points expressed in coordinates (x, z), and the distance d between two points in the set of points may be i Can fall within the range of 1 to 5 meters. In one embodiment, the processing unit 110 can use two polynomials, such as a left road polynomial and a right road polynomial, to construct an initial vehicle path. The processing unit 110 can calculate the geometric midpoint between the two polynomials and offset each point included in the resulting vehicle path by a predetermined offset (e.g., a smart lane offset), if any (a zero offset can correspond to driving in the middle of the lane). The offset can be in a direction perpendicular to the road segment between any two points in the vehicle path. In another embodiment, the processing unit 110 can use a polynomial and an estimated lane width to offset each point of the vehicle path by half the estimated lane width plus a predetermined offset (e.g., a smart lane offset).

[0151] At step 572, the processing unit 110 may update the vehicle path constructed at step 570. The processing unit 110 may reconstruct the vehicle path constructed at step 570 using a higher resolution so that the d between two points in the set of points representing the vehicle path is k Less than the distance d as mentioned above i For example, the distance dk may fall within the range of 0.1 to 0.3 meters. The processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm, which may produce a cumulative distance vector S corresponding to the total length of the vehicle path (ie, based on the set of points representing the vehicle path).

[0152] At step 574, the processing unit 110 may determine a look-ahead point (expressed in coordinates (x l , z l )). The processing unit 110 can extract a foresight point from the accumulated distance vector S, and the foresight point can be associated with a foresight distance and a foresight time. The foresight distance can have a lower limit ranging from 10 meters to 20 meters and can be calculated as the product of the speed of the vehicle 200 and the foresight time. For example, as the speed of the vehicle 200 decreases, the foresight distance can also decrease (e.g., until it reaches the lower limit). The foresight time can range from 0.5 to 1.5 seconds and can be inversely proportional to the gain of one or more control loops such as a heading error tracking control loop associated with causing a navigation response in the vehicle 200. For example, the gain of the heading error tracking control loop can depend on the bandwidth of a yaw rate loop, a steering actuator loop, vehicle lateral dynamics, etc. Therefore, the higher the gain of the heading error tracking control loop, the shorter the foresight time.

[0153] At step 576, processing unit 110 may determine a heading error and a yaw rate command based on the foresight point determined at step 574. Processing unit 110 may calculate the arc tangent of the foresight point, such as arctan(x l / z l ) to determine the heading error. The processing unit 110 may determine the yaw rate command as the product of the heading error and the high level control gain. If the look ahead distance is not at the lower limit, the high level control gain may be equal to: (2 / look ahead time). Otherwise, the high level control gain may be equal to: (2×velocity of the vehicle 200 / look ahead distance).

[0154] Fig. 5Fis a flow chart illustrating an exemplary process 500F for determining whether a leading vehicle is changing lanes consistent with the disclosed embodiments. At step 580, processing unit 110 may determine navigation information associated with a leading vehicle (e.g., a vehicle traveling in front of vehicle 200). For example, processing unit 110 may use the above combined Figure 5A and Figure 5B The described techniques can be used to determine the position, velocity (e.g., direction and speed), and / or acceleration of the vehicle ahead. The processing unit 110 can also use the above combined Figure 5E The described techniques determine one or more road polynomials, forward view points (associated with vehicle 200 ), and / or tracking trajectories (eg, a set of points describing a path taken by a leading vehicle).

[0155] At step 582, processing unit 110 may analyze the navigation information determined in step 580. In one embodiment, processing unit 110 may calculate the distance between the tracking trajectory and the road polynomial (e.g., along the trajectory). If the change in this distance along the trajectory exceeds a predetermined threshold (e.g., 0.1 to 0.2 meters on a straight road, 0.3 to 0.4 meters on a moderately curved road, and 0.5 to 0.6 meters on a sharp curve road), processing unit 110 may determine that the leading vehicle is likely to be changing lanes. In the case where multiple vehicles are detected traveling in front of vehicle 200, processing unit 110 may compare the tracking trajectories associated with each vehicle. Based on the comparison, processing unit 110 may determine that a vehicle whose tracking trajectory does not match the tracking trajectories of other vehicles is likely to be changing lanes. Processing unit 110 may additionally compare the curvature of the tracking trajectory (associated with the leading vehicle) with the expected curvature of the road segment in which the leading vehicle is traveling. The expected curvature may be extracted from map data (e.g., data from map database 160), from a road polynomial, from tracking trajectories of other vehicles, from prior knowledge about the road, etc. If the difference between the curvature of the tracking trajectory and the expected curvature of the road segment exceeds a predetermined threshold, processing unit 110 may determine that the leading vehicle is likely to be changing lanes.

[0156] In another embodiment, processing unit 110 may compare the instantaneous position of the leading vehicle to a forward view point (associated with vehicle 200) over a specific time period (e.g., 0.5 to 1.5 seconds). If the distance between the instantaneous position of the leading vehicle and the forward view point changes during the specific time period, and the cumulative sum of the changes exceeds a predetermined threshold (e.g., 0.3 to 0.4 meters on a straight road, 0.7 to 0.8 meters on a moderately curved road, and 1.3 to 1.7 meters on a sharp curve road), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the tracking trajectory by comparing the lateral distance traveled along the tracking trajectory with the expected curvature of the tracking path. The expected radius of curvature may be determined based on the calculation: (δ z 2 +δ x 2 ) / 2 / (δ x ), where δ x represents the lateral travel distance and δ z The longitudinal distance traveled is represented by the curvature. If the difference between the lateral distance traveled and the expected curvature exceeds a predetermined threshold (e.g., 500 to 700 meters), the processing unit 110 can determine that the leading vehicle is likely to be changing lanes. In another embodiment, the processing unit 110 can analyze the position of the leading vehicle. If the position of the leading vehicle obscures the road polynomial (e.g., the leading vehicle covers the top of the road polynomial), the processing unit 110 can determine that the leading vehicle is likely to be changing lanes. In the case where the position of the leading vehicle is such that another vehicle is detected in front of the leading vehicle and the tracking trajectories of the two vehicles are not parallel, the processing unit 110 can determine that the (closer) leading vehicle is likely to be changing lanes.

[0157] At step 584, processing unit 110 may determine whether leading vehicle 200 is changing lanes based on the analysis performed at step 582. For example, processing unit 110 may make this determination based on a weighted average of the various analyses performed at step 582. Under such a scheme, for example, a determination made by processing unit 110 that a leading vehicle is likely to be changing lanes based on a particular type of analysis may be assigned a value of "1" (and "0" used to represent a determination that a leading vehicle is unlikely to be changing lanes). Different analyses performed at step 582 may be assigned different weights, and the disclosed embodiments are not limited to any particular combination of analyses and weights.

[0158] Figure 66 is a flow chart illustrating an exemplary process 600 for causing one or more navigation responses based on stereo image analysis consistent with the disclosed embodiments. At step 610, the processing unit 110 may receive the first and second pluralities of images via the data interface 128. For example, a camera included in the image acquisition unit 120 (such as the image capture devices 122 and 124 having the fields of view 202 and 204) may capture the first and second pluralities of images of the area in front of the vehicle 200 and transmit them to the processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, the processing unit 110 may receive the first and second pluralities of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0159] At step 620, the processing unit 110 may execute the stereo image analysis module 404 to perform stereo image analysis on the first and second plurality of images to create a 3D map of the road ahead of the vehicle and detect features within the image, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. Figures 5A to 5D Stereoscopic image analysis may be performed in a manner similar to the steps described above. For example, the processing unit 110 may execute the stereoscopic image analysis module 404 to detect candidate objects (e.g., vehicles, pedestrians, road signs, traffic lights, road hazards, etc.) within the first and second multiple images, filter out a subset of candidate objects based on various criteria, and perform multi-frame analysis, build measurements, and determine confidence levels for the remaining candidate objects. In performing the above steps, the processing unit 110 may consider information from both the first and second multiple images, rather than only considering information from one set of images. For example, the processing unit 110 may analyze the differences in pixel-level data (or other subsets of data from two streams of captured images) of candidate objects that appear in both the first and second multiple images. As another example, the processing unit 110 may estimate the position and / or velocity of the candidate object (e.g., relative to the vehicle 200) by observing that the object appears in one of the multiple images but not in the other image, or other differences that may exist relative to the objects that appear in the two image streams. For example, the position, velocity, and / or acceleration relative to the vehicle 200 may be determined based on features such as trajectory, position, movement characteristics, etc. associated with the object that appears in one or both of the image streams.

[0160] At step 630, the processing unit 110 may execute the navigation response module 408 to perform the navigation response based on the analysis performed at step 620 and the above combination. Figure 4The described techniques may cause one or more navigation responses in the vehicle 200. The navigation responses may include, for example, steering, lane changes, changes in acceleration, changes in speed, braking, etc. In some embodiments, the processing unit 110 may cause the one or more navigation responses using data derived from executing the speed and acceleration module 406. Furthermore, multiple navigation responses may occur simultaneously, sequentially, or any combination thereof.

[0161] Figure 7 7 is a flow chart showing an exemplary process 700 for causing one or more navigation responses based on analysis of three sets of images consistent with the disclosed embodiments. At step 710, the processing unit 110 may receive the first, second, and third plurality of images via the data interface 128. For example, a camera included in the image acquisition unit 120 (such as the image capture devices 122, 124, and 126 having the fields of view 202, 204, and 206) may capture the first, second, and third plurality of images of the area in front of and / or to the side of the vehicle 200 and transmit them to the processing unit 110 via a digital connection (e.g., USB, wireless, Bluetooth, etc.). In some embodiments, the processing unit 110 may receive the first, second, and third plurality of images via three or more data interfaces. For example, each of the image capture devices 122, 124, 126 may have an associated data interface for communicating data to the processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

[0162] At step 720, processing unit 110 may analyze the first, second, and third plurality of images to detect features within the images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, road hazards, etc. This analysis may be performed in a manner similar to the above combined Figure 5A-5D and Figure 6 For example, the processing unit 110 may perform a monocular image analysis on each of the first, second, and third plurality of images (e.g., via execution of the monocular image analysis module 402 and based on the above combination FIG. 5A to FIG. 5D Alternatively, the processing unit 110 may perform stereoscopic image analysis on the first and second plurality of images, the second and third plurality of images, and / or the first and third plurality of images (e.g., via execution of the stereoscopic image analysis module 404 and based on the above combined Figure 64 and 5. The processing unit 110 may perform a combination of monocular and stereo image analysis. For example, the processing unit 110 may perform monocular image analysis on the first plurality of images (e.g., via execution by the monocular image analysis module 402) and stereo image analysis on the second and third plurality of images (e.g., via execution by the stereo image analysis module 404). The configuration of the image capture devices 122, 124, and 126 - including their respective positioning and fields of view 202, 204, and 206 - may affect the type of analysis performed on the first, second, and third plurality of images. The disclosed embodiments are not limited to a particular configuration of the image capture devices 122, 124, and 126 or the type of analysis performed on the first, second, and third plurality of images.

[0163] In some embodiments, processing unit 110 may test system 100 based on the images acquired and analyzed at steps 710 and 720. Such testing may provide an indicator of the overall performance of system 100 for certain configurations of image acquisition devices 122, 124, and 126. For example, processing unit 110 may determine the proportion of "false hits" (e.g., instances where system 100 incorrectly determines the presence of a vehicle or pedestrian) and "misses."

[0164] At step 730, the processing unit 110 may cause one or more navigation responses in the vehicle 200 based on information derived from two of the first, second, and third plurality of images. The selection of two of the first, second, and third plurality of images may depend on various factors, such as, for example, the number, type, and size of objects detected in each of the plurality of images. The processing unit 110 may also make a selection based on image quality and resolution, the effective field of view reflected in the image, the number of frames captured, the degree to which one or more objects of interest actually appear in the frame (e.g., the percentage of frames in which the object appears, the proportion of the object appearing in each such frame), etc.

[0165] In some embodiments, processing unit 110 may select information derived from two of the first, second, and third pluralities of images by determining the degree to which information derived from one image source is consistent with information derived from other image sources. For example, processing unit 110 may combine processed information derived from each of image capture devices 122, 124, and 126 (whether through monocular analysis, stereo analysis, or any combination of both) and determine visual indicators (e.g., lane markings, detected vehicles and their locations and / or paths, detected traffic lights, etc.) that are consistent between each captured image from image capture devices 122, 124, and 126. Processing unit 110 may also exclude information that is inconsistent between captured images (e.g., vehicles changing lanes, lane models indicating that vehicles are too close to vehicle 200, etc.). Thus, processing unit 110 may select information derived from two of the first, second, and third pluralities of images based on determining consistent and inconsistent information.

[0166] The navigation response may include, for example, steering, lane changing, braking, acceleration change, etc. The processing unit 110 may make a decision based on the analysis performed in step 720 and the above combination. Figure 4 The processing unit 110 may also cause one or more navigation responses using the techniques described herein. The processing unit 110 may also cause one or more navigation responses using data derived from executing the speed and acceleration module 406. In some embodiments, the processing unit 110 may cause one or more navigation responses based on the relative position, relative velocity, and / or relative acceleration between the vehicle 200 and an object detected within any of the first, second, and third pluralities of images. The multiple navigation responses may occur simultaneously, sequentially, or any combination thereof.

[0167] Analysis of the captured images may allow the generation and use of a sparse map model for autonomous vehicle navigation. In addition, analysis of the captured images may allow the autonomous vehicle to be localized using the identified lane markings. Fig. 20 , embodiments for detecting specific features based on one or more pending analyses of captured images, and for navigating an autonomous vehicle using a sparse map model are discussed.

[0168] Sparse road models for autonomous vehicle navigation

[0169] In some embodiments, the disclosed systems and methods may use sparse maps for autonomous vehicle navigation. In particular, sparse maps may be used for autonomous vehicle navigation along road segments. For example, sparse maps may provide sufficient information for navigating an autonomous vehicle without storing and / or updating a large amount of data. As discussed in more detail below, an autonomous vehicle may use a sparse map to navigate one or more roads based on one or more stored trajectories.

[0170] Sparse maps for autonomous vehicle navigation

[0171] In some embodiments, the disclosed systems and methods can generate sparse maps for autonomous vehicle navigation. For example, sparse maps can provide sufficient information for navigation without requiring excessive data storage or data transfer rates. As discussed in more detail below, a vehicle (which can be an autonomous vehicle) can use a sparse map to navigate one or more roads. For example, in some embodiments, a sparse map can contain data related to roads and potential landmarks along the roads that may be sufficient for vehicle navigation, but also exhibit a small data footprint. For example, the sparse data maps described in detail below may require significantly less storage space and data transfer bandwidth than digital maps containing detailed map information (such as image data collected along roads).

[0172] For example, a sparse data map can store a three-dimensional polynomial representation of a preferred vehicle path along a road, rather than storing a detailed representation of a road segment. These paths may require very little data storage space. In addition, in the described sparse data map, landmarks can be identified and included in a sparse map road model to help navigation. These landmarks can be located at any spacing suitable for enabling vehicle navigation, but in some cases, these landmarks do not need to be identified and included in the model with high density and short spacing. On the contrary, in some cases, navigation based on landmarks with a spacing of at least 50 meters, at least 100 meters, at least 500 meters, at least 1 kilometer or at least 2 kilometers is possible. As will be discussed in more detail in other chapters, a sparse map can be generated based on data collected or measured by a vehicle equipped with various sensors and devices (such as image capture devices, global positioning system sensors, motion sensors, etc.) when the vehicle travels along a road. In some cases, a sparse map can be generated based on data collected during multiple driving of one or more vehicles along a specific road. Using multiple driving of one or more vehicles to generate a sparse map can be referred to as a "crowdsourcing" sparse map.

[0173] Consistent with the disclosed embodiments, an autonomous vehicle system may use a sparse map for navigation. For example, the disclosed systems and methods may allocate a sparse map for generating a road navigation model for an autonomous vehicle, and may use the sparse map and / or the generated road navigation model to navigate the autonomous vehicle along a road segment. A sparse map consistent with the present disclosure may include one or more three-dimensional contours that may represent predetermined trajectories that autonomous vehicles may traverse as they move along an associated road segment.

[0174] Sparse maps consistent with the present disclosure may also include data representing one or more road features. Such road features may include recognizable landmarks, road signature profiles, and any other road-related features useful in vehicle navigation. Sparse maps consistent with the present disclosure may enable autonomous navigation of a vehicle based on a relatively small amount of data contained in the sparse map. For example, the disclosed embodiments of the sparse map may require a relatively small storage space (and a relatively small bandwidth when portions of the sparse map are transmitted to the vehicle), but may still adequately provide autonomous vehicle navigation, rather than including detailed representations of roads, such as road edges, road curvature, images associated with road segments, or data detailing other physical features associated with road segments. In some embodiments, the small data footprint of the disclosed sparse map may be achieved by storing representations of road-related elements that require a small amount of data but still enable autonomous navigation, which will be discussed in further detail below.

[0175] For example, the disclosed sparse map may store a polynomial representation of one or more trajectories that a vehicle may follow along a road, rather than storing a detailed representation of various aspects of a road. Thus, using the disclosed sparse map, a vehicle may be navigated along a particular road segment, rather than storing (or having to transmit) details about the physical properties of the road to enable navigation along the road, in some cases without having to account for the physical aspects of the road, but rather by aligning the path it travels with a trajectory (e.g., a polynomial spline) along a particular road segment. In this way, a vehicle may be navigated primarily based on a stored trajectory (e.g., a polynomial spline), which may require significantly less storage space than methods involving storage of road images, road parameters, road layouts, etc.

[0176] In addition to the stored polynomial representations of trajectories along road segments, the disclosed sparse maps may also contain small data objects that may represent road features. In some embodiments, the small data objects may contain digital signatures derived from digital images (or digital signals) obtained by sensors (e.g., cameras or other sensors, such as suspension sensors) on vehicles traveling along the road segment. The digital signatures may have a reduced size relative to the signals acquired by the sensors. In some embodiments, the digital signatures may be created to be compatible with a classifier function that is configured to detect and identify road features from signals acquired by the sensors, for example, during subsequent driving. In some embodiments, digital signatures may be created so that the digital signatures have as small a footprint as possible while retaining the ability to associate or match road features with stored signatures based on images of road features (or digital signals generated by sensors if the stored signatures are not image-based and / or contain other data) captured by cameras on subsequent vehicles traveling along the same road segment.

[0177] In some embodiments, the size of the data object may be further associated with the uniqueness of the road feature. For example, for a road feature that is detectable by a camera on a vehicle, and where the camera system on the vehicle is coupled to a classifier that is capable of distinguishing image data corresponding to the road feature as being associated with a particular type of road feature (e.g., a road sign), and where such a road sign is locally unique in the area (e.g., there are no identical road signs or road signs of the same type nearby), it may be sufficient to store data indicating the type of road feature and its location.

[0178] As will be discussed in further detail below, road features (e.g., landmarks along road segments) can be stored as small data objects that can represent road features with relatively few bytes while providing enough information for identifying and navigating using such features. In one example, road signs can be identified as recognizable landmarks on which navigation of a vehicle can be based. Representations of road signs can be stored in sparse maps to include, for example, a few bytes of data indicating the type of landmark (e.g., a stop sign) and a few bytes of data indicating the location of the landmark (e.g., coordinates). Navigation based on such a data-light representation of landmarks (e.g., using a representation sufficient to locate, identify, and navigate based on landmarks) can provide the desired level of navigation functionality associated with sparse maps without significantly increasing the data overhead associated with sparse maps. Such a condensed representation of landmarks (and other road features) can utilize sensors and processors configured to detect, identify, and / or classify specific road features contained on such vehicles.

[0179] For example, when a landmark or even a particular type of landmark in a given area is locally unique (e.g., when there are no other landmarks or no other landmarks of the same type), the sparse map can use data indicating the type of landmark (the landmark or the particular type of landmark), and during navigation (e.g., autonomous navigation) when a camera on an autonomous vehicle captures an image of an area containing the landmark (or the particular type of landmark), the processor can process the image, detect the landmark (if indeed present in the image), classify the image as the landmark (or the particular type of landmark), and associate the location of the image with the location of the landmark stored in the sparse map.

[0180] Road feature representation

[0181] In some embodiments, the sparse map may include at least one line representation of road surface features extending along a road segment and a plurality of landmarks associated with the road segment. In some aspects, the sparse map may be generated via "crowd sourcing", e.g., by performing image analysis on a plurality of images acquired as one or more vehicles traverse the road segment.

[0182] In addition to target trajectories and identified landmarks, sparse maps can contain information related to various other road features. For example, Fig. 8A A representation of a curve along a particular road segment that may be stored in a sparse map is shown. In some embodiments, a single lane of a road may be modeled by a three-dimensional polynomial description of the left and right sides of the road. Fig. 8A Such polynomials for the left and right sides of a single lane are shown in . Regardless of how many lanes a road may have, a road may be represented by a polynomial similar to Fig. 8A For example, the left and right sides of a multi-lane road can be represented by polynomials similar to Fig. 8A The polynomial shown in FIG. 1 and the intermediate lane markings (e.g., dashed line markings indicating lane boundaries, solid yellow lines indicating boundaries between lanes traveling in different directions, etc.) included on multi-lane roads may also be represented by a polynomial such as Fig. 8A The polynomial shown is used to represent .

[0183] like Fig. 8A As shown, a polynomial (e.g., a first order, second order, third order, or any suitable order polynomial) may be used to represent lane 800. For illustration, lane 800 is shown as a two-dimensional lane, and the polynomial is shown as a two-dimensional polynomial. Fig. 8A As shown, lane 800 includes a left side 810 and a right side 820. In some embodiments, more than one polynomial may be used to represent the location of each side of a road or lane boundary. For example, each of the left side 810 and the right side 820 may be represented by a plurality of polynomials of any suitable length. In some cases, the polynomial may have a length of about 100 m, although other lengths greater than or less than 100 m may also be used. In addition, the polynomials may overlap with each other so as to facilitate seamless transitions in navigation based on subsequent polynomials encountered as the host vehicle travels along the road. For example, each of the left side 810 and the right side 820 may be represented by a plurality of third-order polynomials that are divided into segments of about 100 meters in length (an example of a first predetermined range) and overlap each other by about 50 meters. The polynomials representing the left side 810 and the right side 820 may or may not have the same order. For example, in some embodiments, some polynomials may be second-order polynomials, some may be third-order polynomials, and some may be fourth-order polynomials.

[0184] exist Fig. 8A In the example shown, the left side 810 of lane 800 is represented by two sets of third-order polynomials. The first set contains polynomial segments 811, 812, and 813. The second set contains polynomial segments 814, 815, and 816. The two sets, while being substantially parallel to each other, follow the positioning of their respective road sides. Polynomial segments 811, 812, 813, 814, 815, and 816 have a length of approximately 100 meters and overlap with adjacent segments in the series by approximately 50 meters. However, as previously described, polynomials of different lengths and different amounts of overlap may also be used. For example, the polynomial may have a length of 500m, 1km, or longer, and the amount of overlap may vary from 0 to 50m, 50m to 100m, or more than 100m. In addition, although Fig. 8A The polynomials are shown as representing polynomials extending in 2D space (e.g., on the surface of paper), but it will be appreciated that these polynomials may represent curves extending in three dimensions (e.g., containing a height component) to represent elevation changes in a road segment in addition to XY curvature. Fig. 8A In the example shown, the right side 820 of lane 800 is further represented by a first group having polynomial segments 821 , 822 , and 823 and a second group having polynomial segments 824 , 825 , and 826 .

[0185] Returning to the target trajectory of the sparse map, Figure 8B 3D polynomials representing target trajectories for a vehicle traveling along a particular road segment are shown. The target trajectory represents not only the XY path that the host vehicle should travel along the particular road segment, but also the elevation changes that the host vehicle will experience while traveling along the road segment. Therefore, each target trajectory in the sparse map can be represented by one or more 3D polynomials, such as Figure 8B The three-dimensional polynomial 850 shown in FIG. A sparse map may contain multiple trajectories (e.g., millions or billions or more to represent the trajectories of vehicles along various road segments along roads around the world). In some embodiments, each target trajectory may correspond to a spline connecting the three-dimensional polynomial segments.

[0186] Regarding the data footprint of the polynomial curves stored in the sparse map, in some embodiments, each cubic polynomial can be represented by four parameters, each requiring four bytes of data. A suitable representation can be obtained using a cubic polynomial requiring approximately 192 bytes of data per 100m. For a host vehicle traveling approximately 100km / hr, this can translate into a data usage / transmission requirement of approximately 200kB per hour.

[0187] Sparse maps can use a combination of geometry descriptors and metadata to describe the lane network. The geometry can be described using polynomials or splines as described above. The metadata can describe the number of lanes, special features (such as shared lanes), and possibly other sparse labels. The total footprint of such indicators is likely to be negligible.

[0188] Accordingly, a sparse map according to an embodiment of the present disclosure may include at least one line representation of a road surface feature extending along a road segment, each line representation representing a path along the road segment that substantially corresponds to the road surface feature. In some embodiments, as described above, at least one line representation of a road surface feature may include a spline, a polynomial representation, or a curve. Furthermore, in some embodiments, the road surface feature may include at least one of a road edge or a lane marking. Furthermore, as discussed below with respect to "crowdsourcing," road surface features may be identified by image analysis of multiple images acquired as one or more vehicles traverse a road segment.

[0189] Fig.9A A polynomial representation of a trajectory captured in the process of building or maintaining a sparse map is shown. The polynomial representation of a target trajectory contained in the sparse map can be determined based on two or more reconstructed trajectories previously traversed by a vehicle along the same road segment. In some embodiments, the polynomial representation of the target trajectory contained in the sparse map can be an aggregation of two or more reconstructed trajectories previously traversed by a vehicle along the same road segment. In some embodiments, the polynomial representation of the target trajectory contained in the sparse map can be an average of two or more reconstructed trajectories previously traversed by a vehicle along the same road segment. Other mathematical operations can also be used to construct a target trajectory along a road path based on reconstructed trajectories collected from vehicles traversing along a road segment.

[0190] like Fig.9A As shown, a road segment 900 can be driven by multiple vehicles 200 at different times. Each vehicle 200 can collect data related to the path taken by the vehicle along the road segment. The path driven by a particular vehicle can be determined based on camera data, accelerometer information, speed sensor information and / or GPS information and other potential sources. Such data can be used to reconstruct the trajectory of the vehicle traveling along the road segment, and based on these reconstructed trajectories, a target trajectory (or multiple target trajectories) can be determined for a particular road segment. Such a target trajectory can represent the preferred path of the vehicle when the host vehicle travels along the road segment (e.g., guided by an autonomous navigation system).

[0191] exist Fig.9AIn the example shown, a first reconstructed trajectory 901 may be determined based on data received from a first vehicle that traverses the road segment 900 in a first time period (e.g., day 1), a second reconstructed trajectory 902 may be obtained from a second vehicle that traverses the road segment 900 in a second time period (e.g., day 2), and a third reconstructed trajectory 903 may be obtained from a third vehicle that traverses the road segment 900 in a third time period (e.g., day 3). Each trajectory 901, 902, and 903 may be represented by a polynomial, such as a three-dimensional polynomial. It should be noted that in some embodiments, any reconstructed trajectory may be assembled on a vehicle that traverses the road segment 900.

[0192] Additionally or alternatively, such a reconstructed trajectory may be determined on the server side based on information received from vehicles traversing the road segment 900. For example, in some embodiments, the vehicles 200 may transmit data related to their movement along the road segment 900 (e.g., steering angle, heading, time, location, speed, sensed road geometry and / or sensed landmarks, etc.) to one or more servers. The server may reconstruct the trajectory of the vehicle 200 based on the received data. The server may also generate a target trajectory for guiding navigation of an autonomous vehicle that will travel along the same road segment 900 at a later time based on the first trajectory 901, the second trajectory 902, and the third trajectory 903. Although a target trajectory may be associated with a single previous traversal of a road segment, in some embodiments, each target trajectory included in the sparse map may be determined based on two or more reconstructed trajectories of vehicles traversing the same road segment. Fig.9A In the sparse map, the target track is represented by 910. In some embodiments, the target track 910 can be generated based on the average of the first track 901, the second track 902, and the third track 903. In some embodiments, the target track 910 included in the sparse map can be an aggregation (e.g., a weighted combination) of two or more reconstructed tracks. The following discusses aligning driving data to construct tracks.

[0193] Fig. 9B and Fig. 9C The concept of target trajectories associated with road segments existing within the geographic area 911 is further illustrated. Fig. 9BAs shown, a first road segment 920 within geographic area 911 may include a multi-lane road including two lanes 922 designated for vehicles traveling in a first direction and two additional lanes 924 designated for vehicles traveling in a second direction opposite to the first direction. Lanes 922 and lanes 924 may be separated by double yellow lines 923. Geographic area 911 may also include a branch road segment 930 intersecting road segment 920. Road segment 930 may include a two-lane road, each lane being designated for a different direction of travel. Geographic area 911 may also include other road features, such as stop lines 932, stop signs 934, speed limit signs 936, and hazard signs 938.

[0194] like Fig. 9C As shown, the sparse map may include a local map 940 that includes a road model for assisting in autonomous navigation of a vehicle within geographic area 911. For example, local map 940 may include target trajectories for one or more lanes associated with road segments 920 and / or 930 within geographic area 911. For example, local map 940 may include target trajectories 941 and / or 942 that the autonomous vehicle may access or rely on when crossing lane 922. Similarly, local map 940 may include target trajectories 943 and / or 944 that the autonomous vehicle may access or rely on when crossing lane 924. In addition, local map 940 may include target trajectories 945 and / or 946 that the autonomous vehicle may access or rely on when crossing road segment 930. Target trajectory 947 represents a preferred path that the autonomous vehicle should follow when transitioning from lane 920 (specifically, relative to target trajectory 941 associated with the rightmost lane of lane 920) to road segment 930 (specifically, relative to target trajectory 945 associated with the first side of road segment 930). Similarly, target trajectory 948 represents a preferred path that the autonomous vehicle should follow when transitioning from road segment 930 (specifically, relative to target trajectory 946) to a portion of road segment 924 (specifically, as shown, relative to target trajectory 943 associated with the left lane of lane 924).

[0195] The sparse map may also include representations of other road-related features associated with geographic area 911. For example, the sparse map may also include representations of one or more landmarks identified in geographic area 911. These landmarks may include a first landmark 950 associated with stop line 932, a second landmark 952 associated with stop sign 934, a third landmark associated with speed limit sign 954, and a fourth landmark 956 associated with hazard sign 938. Such landmarks may be used, for example, to help the autonomous vehicle determine its current location relative to any shown target track so that the vehicle can adjust its heading to match the direction of the target track at the determined location.

[0196] In some embodiments, the sparse map may also include road signature profiles. Such road signature profiles may be associated with any discernible / measurable change in at least one parameter associated with a road. For example, in some cases, such profiles may be associated with changes in road surface information, such as changes in surface roughness of a particular road segment, changes in road width on a particular road segment, changes in distance between dashed lines drawn along a particular road segment, changes in road curvature along a particular road segment, etc. Fig.9D An example of a road signature profile 960 is shown. While profile 960 may represent any of the above parameters or other parameters, in one example, profile 960 may represent a measurement of road surface roughness, e.g., as obtained by monitoring one or more sensors that provide outputs indicative of an amount of suspension displacement as a vehicle travels on a particular road segment.

[0197] Alternatively or simultaneously, profile 960 can represent changes in road width, as determined based on image data obtained via a camera on a vehicle traveling on a particular road segment. For example, such a profile is useful in determining a particular positioning of an autonomous vehicle relative to a particular target track. That is, as it traverses a road segment, the autonomous vehicle can measure a profile associated with one or more parameters associated with the road segment. If the measured profile can be correlated / matched with a predetermined profile that maps parameter changes relative to a position along the road segment, the measured and predetermined profiles can be used (e.g., by overlaying corresponding portions of the measured and predetermined profiles) to determine a current position along the road segment, and therefore determine a current position of the target track relative to the road segment.

[0198] In some embodiments, the sparse map may include different trajectories based on different characteristics associated with users of the autonomous vehicle, environmental conditions, and / or other parameters related to driving. For example, in some embodiments, different trajectories may be generated based on different user preferences and / or profiles. Sparse maps containing such different trajectories may be provided to different autonomous vehicles of different users. For example, some users may prefer to avoid toll roads, while other users may prefer to take the shortest or fastest route, regardless of whether there are toll roads on the route. The disclosed system may generate different sparse maps with different trajectories based on such different user preferences or profiles. As another example, some users may prefer to drive in a fast-moving lane, while other users may prefer to always maintain a position in the center lane.

[0199] Based on different environmental conditions, such as day and night, snow, rain, fog, etc., different trajectories can be generated and included in the sparse map. Autonomous vehicles traveling under different environmental conditions can be provided with sparse maps generated based on such different environmental conditions. In some embodiments, a camera provided on an autonomous vehicle can detect environmental conditions, and such information can be provided back to a server that generates and provides a sparse map. For example, a server can generate or update an already generated sparse map to include trajectories that may be more suitable or safer for autonomous driving under detected environmental conditions. As the autonomous vehicle travels along the road, updates to the sparse map based on environmental conditions can be performed dynamically.

[0200] Other different parameters related to driving can also be used as a basis for generating different sparse maps and providing different sparse maps to different autonomous vehicles. For example, when an autonomous vehicle is traveling at high speeds, turns may be tighter. Tracks associated with specific lanes rather than roads can be included in the sparse map so that the autonomous vehicle can maintain itself within a specific lane when the vehicle follows a specific track. When images captured by a camera on the autonomous vehicle indicate that the vehicle has drifted outside of the lane (e.g., crossed a lane marking), an action can be triggered within the vehicle to bring the vehicle back to the designated lane according to the specific track.

[0201] Fig.10 An example autonomous vehicle road navigation model represented by a plurality of three-dimensional splines 1001 , 1002 , and 1003 is shown. Fig.10 The curves 1001, 1002, and 1003 shown in are for illustration purposes only. Each spline may include one or more three-dimensional polynomials connecting a plurality of data points 1010. Each polynomial may be a first-order polynomial, a second-order polynomial, a third-order polynomial, or a combination of any suitable polynomials of different orders. Each data point 1010 may be associated with navigation information received from a plurality of vehicles. In some embodiments, each data point 1010 may be associated with data associated with a landmark (e.g., the size, location, and identification information of the landmark) and / or a road signature profile (e.g., road geometry, road roughness profile, road curvature profile, road width profile). In some embodiments, some data points 1010 may be associated with data associated with a landmark, while other data points may be associated with data associated with a road signature profile.

[0202] Fig.11The raw positioning data 1110 (e.g., GPS data) received from five separate drivings are shown. If a driving is simultaneously separated by a vehicle at the same time, by the same vehicle at a separated time, or by a separated vehicle at a separated time, the driving can be separated from another driving. In order to take into account the errors in the positioning data 1110 and the different positioning of vehicles in the same lane (e.g., one vehicle may be closer to the left side of the lane than another vehicle), the remote server can use one or more statistical techniques to generate a map skeleton 1120 to determine whether the changes in the raw positioning data 1110 represent actual deviations or statistical errors. Each path in the skeleton 1120 can be linked back to the raw data 1110 that formed the path. For example, the path between A and B in the skeleton 1120 is linked to the raw data 1110 from driving 2, 3, 4, and 5 instead of from driving 1. The skeleton 1120 may not be detailed enough for navigating the vehicle (e.g., because unlike the above-mentioned spline, the skeleton 1120 combines the driving of multiple lanes on the same road), but it can provide useful topological information and can be used to define intersections.

[0203] Safety and comfort constraints for navigation

[0204] In addition to navigation models, autonomous vehicles (whether fully autonomous, such as self-driving vehicles, or partially autonomous, such as one or more driver assistance systems or features) typically use driving strategies to ensure the safety of other drivers and pedestrians and the comfort of interior passengers.

[0205] Therefore, the autonomous vehicle can sense the navigation state in the environment of the host vehicle. For example, the vehicle can rely on inputs from various sensors and sensing systems associated with the host vehicle. These inputs can include images or image streams from one or more on-board cameras, GPS location information, accelerometer output, user feedback, or user input to one or more user interface devices, radar, laser radar, etc. Sensing (which can include data from cameras and / or any other available sensors and map information) can be collected, analyzed and formed into a "sensed state" that describes the information extracted from the scene in the environment of the host vehicle. The sensed state can include sensed information related to target vehicles, lane markings, pedestrians, traffic lights, road geometry, lane shapes, obstacles, distances to other objects / vehicles, relative speeds, relative accelerations, and any other potential sensed information. Supervised machine learning can be implemented to generate a sensed state output based on the sensed data provided. As described below, the output of the sensing module can represent the sensed navigation "state" of the host vehicle, which can be used in the driving strategy.

[0206] Although the sensed state can be generated based on image data received from one or more cameras or image sensors associated with the main vehicle, any suitable sensor or sensor combination can be used to generate the sensed state for navigation. In some embodiments, the sensed state can be generated independently of the captured image data. In fact, any navigation principle described herein can be applied to sensed states generated based on captured image data and sensed states generated using other non-image-based sensors. The sensed state can also be determined via a source outside the main vehicle. For example, the sensed state can be generated based on information received from a source away from the main vehicle (e.g., based on sensor information shared from other vehicles, shared from a central server, or from any other information source related to the navigation state of the main vehicle, processed state information, etc.).

[0207] The autonomous vehicle can implement a desired driving strategy to determine one or more navigation actions to be taken by the host vehicle in response to the sensed navigation state. If there are no other agents (e.g., target vehicles or pedestrians) in the environment of the host vehicle, the sensed state input can be processed in a relatively direct manner. When the sensed state needs to be negotiated with one or more other actors, the task becomes more complicated. Techniques for generating the output of the driving strategy may include reinforcement learning (discussed in more detail below). The output of the driving strategy may include at least one navigation action of the host vehicle, and may include a desired acceleration (which can be converted into an updated speed of the host vehicle), a desired yaw rate of the host vehicle, a desired trajectory, and other potential desired navigation actions.

[0208] Based on the output from the driving strategy, the autonomous vehicle can generate control instructions for one or more actuators or controlled devices associated with the host vehicle. Such actuators and devices can include accelerators, one or more steering controllers, brakes, signal transmitters, displays, or any other actuators or devices that can be controlled as part of the navigation operation associated with the host vehicle. Various aspects of control theory can be used to generate control instructions. The instructions for the controllable components of the host vehicle can achieve the desired navigation goals or requirements of the driving strategy.

[0209] Returning to the driving strategy discussed above, in some embodiments, a trained system trained by reinforcement learning can be used to implement the driving strategy. In other embodiments, the driving strategy can be implemented without machine learning methods by using a specified algorithm to "manually" solve various scenarios that may arise during autonomous navigation. However, while this approach is feasible, it may lead to oversimplification of the driving strategy and may lack the flexibility of a trained system based on machine learning. For example, a trained system may be better equipped to handle complex navigation states and may better determine whether a taxi is parking or stopping to pick up / drop off passengers; determine whether a pedestrian intends to cross the street in front of the main vehicle; defensively balance unexpected behavior of other drivers; negotiate in dense traffic involving target vehicles and / or pedestrians; decide when to suspend certain navigation rules or augment other rules; anticipate unsensed but expected situations (e.g., whether a pedestrian will emerge from behind a car or obstacle); and so on. A trained system based on reinforcement learning may also be better equipped to solve continuous and high-dimensional state spaces and continuous action spaces.

[0210] Training the system using reinforcement learning can involve learning a driving policy to map from sensed states to navigation actions. The driving policy can be a function π:S→A, where S is a set of states and is the action space (e.g., desired velocity, acceleration, yaw command, etc.). State space S = S s ×S p , where S s is the sensing state, and S p is additional information about the state saved by the strategy. Working in discrete time intervals, at time t, the current state s can be observed t ∈S, and the policy can be applied to obtain the desired action a t =π(s t ).

[0211] The system can be trained by exposing it to various navigation states, causing it to apply the policy, and providing rewards (based on a reward function designed to reward the desired navigation behavior). Based on the reward feedback, the system can "learn" the policy and become trained in producing the desired navigation actions. For example, the learning system can observe the current state s t ∈S, and decides action a based on strategy π:S→D(A) t ∈A. Based on the action decided (and the execution of that action), the environment moves to the next state s t+1∈S, for the learning system to observe. For each action taken in response to the observed state, the feedback to the learning system is a reward signal r1,r2,….

[0212] The goal of reinforcement learning (RL) is to find a policy π. It is usually assumed that at time t, there exists a reward function r t , whose measurement is in state s t and take action a t However, taking action a at time t t affects the environment and, therefore, the value of future states. As a result, when deciding which action to take, not only the current reward should be considered, but also future rewards. In some cases, when the system determines that a higher reward may be achieved in the future if a lower reward option is taken now, then the system should take an action even if that action is associated with a lower reward than another available option. To formalize this, observe that the policy π and the initial state s are summarized in R T distribution over , where if the agent starts at state s0 = s and follows policy π from there, then the vector (r1, …, r T ) is the probability of observing rewards r1,…,r T The value of the initial state s can be defined as:

[0213]

[0214] Instead of limiting the time horizon to T, we can discount future rewards to define, for some fixed γ∈(0,1),

[0215]

[0216] In any case, the best strategy is the solution to:

[0217]

[0218] Among them, the expectation is above the initial state s.

[0219] There are several possible approaches for training a driving policy system. For example, one can use imitation methods (e.g., behavioral cloning), in which the system learns from state / action pairs, where the actions are those that a good actor (e.g., a human) would choose in response to a particular observed state. Suppose that a human driver is observed. From this observation, many forms of (s t ,a t ) (where s t is the state, and a tis the action of a human driver) can be obtained, observed, and used as the basis for training a driving policy system. For example, supervised learning can be used to learn a policy π such that π(s t )≈at 。 This approach has many potential advantages. First, it does not require the definition of a reward function. Second, the learning is supervised and occurs offline (no actors need to be applied during the learning process). A disadvantage of this approach is that different human drivers, or even the same human driver, are not deterministic in their choice of strategy. Therefore, for ||π(s t )-a t ||Learning of very small functions is usually not feasible. Moreover, even small errors may accumulate to produce larger errors over time.

[0220] Another technique that can be used is policy-based learning. Here, the policy can be expressed in parameterized form and optimized directly using a suitable optimization technique (e.g., stochastic gradient descent). There are many other ways to solve this problem. One advantage of this approach is that it solves the problem directly, so it often leads to good practical results. A potential disadvantage is that it often requires "online policy" training, that is, the learning of π is an iterative process, where at iteration j, there is a non-perfect policy π j , and in order to construct the next strategy π j , must be based on π j Interact with the environment while performing actions.

[0221] The system can also be trained by value-based learning (learning Q or V functions). Assume that an optimal value function V* can be learned with good approximation. An optimal policy can be constructed (e.g., relying on the Bellman equation). Some versions of value-based learning can be implemented offline (called "off-policy" training). Some disadvantages of value-based methods may be due to their strong reliance on Markovian assumptions and the need to approximate complex functions (which may be more difficult to approximate the value function than to directly approximate the policy).

[0222] Another technique can include model-based learning and planning (learning the probabilities of state transitions and solving the optimization problem of finding the optimal V). A combination of these techniques can also be used to train a learning system. In this approach, the dynamics of the process can be learned, i.e., taking (s t ,a t ) and in the next state s t+1Once this function is learned, the optimization problem can be solved to find the policy π whose value is optimal. This is called "planning". One advantage of this approach may be that the learning is partially supervised and can be obtained by observing the triples (s t ,a t ,s t+1 ) and is applied offline. Similar to the “imitation” approach, a drawback of this approach may be that small errors in the learning process may accumulate and produce a policy that is not properly executed.

[0223] Another approach for training a driving policy module can include decomposing the driving policy function into semantically meaningful components. This allows some parts of the policy to be implemented manually, which can ensure the safety of the policy; and other parts of the policy can be implemented using reinforcement learning techniques, which can be adaptive to many scenarios, human-like balance between defensive / aggressive behavior, and human-like negotiation with other drivers. From a technical perspective, reinforcement learning methods can combine several methods and provide an easy-to-use training procedure, most of which can be performed using recorded data or a self-built simulator.

[0224] In some embodiments, the training of the driving policy module can rely on an "option" mechanism. To illustrate, consider the simple scenario of a driving policy on a two-lane highway. In a direct RL approach, policy π maps states to , where the first component of π(s) is the desired acceleration command and the second component of π(s) is the yaw rate. In the modified approach, the following strategy can be constructed:

[0225] Automatic Cruise Control (ACC) strategy, o ACC :S→A: This strategy always outputs a yaw rate of 0 and only changes the rate to achieve smooth and accident-free driving.

[0226] ACC+left strategy, o L :S→A: The longitudinal commands for this strategy are the same as the ACC commands. The yaw rate is a direct implementation of centering the vehicle towards the middle of the left lane while ensuring safe lateral movement (e.g., not moving to the left if there is a car on the left).

[0227] ACC+Right Strategy, o R :S→A:with o L Same, but the vehicle may center toward the center of the right lane.

[0228] These strategies can be called "options". Based on these "options", we can learn a strategy π that selects the option o :S→O, where O is a set of available options. In one case, O={oACC ,o L ,o R}. Option selector strategy π o By setting To define the actual strategy π:S→A.

[0229] In practice, the policy function can be decomposed into an option graph. The option graph can represent a hierarchical set of decisions organized as a directed acyclic graph (DAG). There is a special node called the root node of the graph. This node has no incoming nodes. The decision process starts from the root node and traverses the entire graph until it reaches a "leaf" node, which refers to a node with no outgoing decision lines. When encountering a leaf node, the driving policy can output acceleration and steering commands associated with the desired navigation action, which is associated with the leaf node.

[0230] An internal node may cause the implementation of a policy that selects children among its available options. The set of available children of an internal node includes all nodes associated with the particular internal node via a decision line.

[0231] Flexibility in the decision-making system can be achieved by enabling nodes to adjust their positions in the hierarchy of the option graph. For example, any node can be allowed to declare itself as "critical". Each node can implement an "is critical" function that outputs "true" if the node is in a critical part of its policy implementation. For example, a node responsible for overtaking can declare itself as critical in the middle of a maneuver. This can impose constraints on the set of available children of node u, which can include all nodes v that are children of node u and for which there is a path from v to a leaf node passing through all nodes designated as critical. On the one hand, this approach can allow the expected path to be declared on the graph at each time step, while on the other hand, the stability of the policy can be retained, especially when the critical part of the policy is being implemented.

[0232] By defining an option graph, the problem of learning a driving policy π: S→A can be decomposed into the problem of defining a policy for each node in the graph, where the policy at the internal node should be selected from the available child nodes. For some nodes, the corresponding policy can be implemented manually (for example, by specifying a set of actions in response to the observed state through an if-then type algorithm), while for other policies, a trained system built by reinforcement learning can be used for implementation. The choice between manual or trained / learned methods can depend on the safety aspects associated with the task and on its relative simplicity. The option graph can be constructed in such a way that some nodes are implemented directly, while other nodes can rely on trained models. This approach can ensure the safe operation of the system.

[0233] As mentioned above, the input to the driving policy is a "sensed state" that summarizes, for example, a map of the environment obtained from available sensors. The output of the driving policy is a set of expectations (optionally together with a set of hard constraints) that define the trajectory as a solution to the optimization problem.

[0234] As described above, the option graph represents a hierarchical decision set organized as a DAG. There is a special node called the "root" of the graph. The root node is the only node with no incoming edges (e.g., decision lines). The decision process traverses the graph starting from the root node until it reaches the "leaf" node, which is a node without outgoing edges. Each internal node should implement a strategy of selecting a child among its available children. Each leaf node should implement a strategy that defines a set of expectations (e.g., a set of navigation goals for the main vehicle) based on the entire path from the root to the leaf. This set of expectations directly defined based on the sensed state together with a set of hard constraints establishes an optimization problem, the solution of which is the trajectory of the vehicle. Hard constraints can be used to further improve the safety of the system, and the expectation can be used to provide the system with driving comfort and human-like driving behavior. The trajectory provided as a solution to the optimization problem further defines the commands that should be provided to the steering, braking and / or engine actuators in order to complete the trajectory.

[0235] Various semantic meanings may be assigned to target vehicles in the host vehicle's environment. For example, in some embodiments, the semantic meaning may include any of the following designations: 1) Not relevant: indicates that the sensed vehicle in the scene is currently not relevant; 2) Next lane: indicates that the sensed vehicle is in an adjacent lane and should maintain an appropriate offset relative to the vehicle (the exact offset may be calculated in an optimization problem that constructs a trajectory given a desired and hard constraint, and the exact offset may potentially be vehicle-dependent - the hold leaf of the option map sets the semantic type of the target vehicle, which defines the desired relative to the target vehicle); 3) Give way: the host vehicle will attempt to give way to the sensed target vehicle by, for example, reducing speed (especially if the host vehicle determines that the target vehicle may cut into the host vehicle's lane); 4) Take way: the host vehicle will attempt to take the right of way by, for example, increasing speed; 5) Follow: the host vehicle expects to follow the target vehicle and maintain a smooth drive; 6) Left / right overtaking: this means that the host vehicle wants to initiate a lane change to the left or right lane.

[0236] Another example of a node is a select gap node. This node may be responsible for selecting a gap between two target vehicles in a particular target lane that the subject vehicle desires to enter. By selecting a node of the form IDj, for some value of j, the subject vehicle arrives at a leaf that specifies a desire for the trajectory optimization problem - for example, the subject vehicle desires to maneuver in order to reach the selected gap. This maneuver may involve first accelerating / braking in the current lane, then heading to the target lane at the appropriate time to enter the selected gap. If the select gap node cannot find a suitable gap, it moves to the abort node, which defines the desire to move back to the center of the current lane and cancel the overtaking.

[0237] As described above, nodes of the option graph can declare themselves as "critical", which can ensure that the selected options pass through critical nodes. Formally, each node can implement an IsCritical function. After performing a forward pass from root to leaf on the option graph, and solving the trajectory planner's optimization problem, a backward pass can be performed from the leaf back to the root. Along this backward pass, the IsCritical function of all nodes in the pass can be called, and a list of all critical nodes can be saved. In the forward path corresponding to the next time frame, the driving policy may be required to choose a path from the root node to the leaf that passes through all critical nodes.

[0238] For example, in the case where an overtaking action is initiated and the driving strategy reaches the leaf corresponding to IDk, such as when the host vehicle is in the middle of an overtaking maneuver, it would be undesirable to select the hold node. To avoid this jump, the IDj node can designate itself as critical. During the maneuver, the success of the trajectory planner can be monitored, and if the overtaking maneuver is performed as expected, the function IsCritical will return a "true" value. This method can ensure that in the next time frame, the overtaking maneuver will continue (rather than jumping to another potentially inconsistent maneuver before completing the initially selected maneuver). On the other hand, if the monitoring of the maneuver indicates that the selected maneuver is not performed as expected, or if the maneuver becomes unnecessary or impossible, the IsCritical function can return a "false" value. This can allow the selection gap node to select a different gap in the next time frame, or to completely abort the overtaking maneuver. This method can, on the one hand, allow the desired path to be declared on the option graph at each time step, while on the other hand, it can help improve the stability of the strategy during the critical part of the execution.

[0239] Hard constraints can be different from navigation expectations, which will be discussed in more detail below. For example, hard constraints can ensure safe driving by applying an additional layer of filtering to planned navigation actions. The hard constraints involved can be determined based on the sensed state, and the hard constraints can be manually programmed and defined, rather than by using a trained system based on reinforcement learning. However, in some embodiments, the trained system can learn the applicable hard constraints to be applied and followed. This approach can prompt the driving strategy module to reach a selected action that already complies with the applicable hard constraints, which can reduce or eliminate the selected action that may need to be modified later to comply with the applicable hard constraints. Nevertheless, as a redundant safety measure, hard constraints can be applied to the output of the driving strategy even when the driving strategy has been trained to cope with predetermined hard constraints.

[0240] There are many potential examples of hard constraints. For example, a hard constraint may be defined in conjunction with a guardrail at the edge of the road. The host vehicle is not allowed to pass the guardrail under any circumstances. Such a rule creates a hard lateral constraint on the trajectory of the host vehicle. Another example of a hard constraint may include a road bump (e.g., a rate control bump), which may cause a hard constraint on the driving speed before the bump and while traversing the bump. Hard constraints may be considered safety critical and therefore may be defined manually rather than relying solely on a trained system that learns the constraints during training.

[0241] In contrast to hard constraints, a desired goal may be to achieve or attain comfortable driving. As discussed above, an example of a desire may include a goal to place the host vehicle within a lane at a lateral position corresponding to the center of the host vehicle's lane. Another desire may include an ID of a gap suitable for entry. Note that the host vehicle need not be precisely in the center of the lane, but rather, a desire to be as close to the center of the lane as possible may ensure that the host vehicle tends to migrate to the center of the lane even when deviating from the center of the lane. Expectations may not be safety critical. In some embodiments, expectations may require negotiation with other drivers and pedestrians. One method of constructing expectations may rely on an option graph, and the policies implemented in at least some nodes of the graph may be based on reinforcement learning.

[0242] For nodes of the option graph implemented as nodes trained based on learning, the training process may include decomposing the problem into a supervised learning phase and a reinforcement learning phase. In the supervised learning phase, it is possible to learn from (s t ,a t )arrive A differentiable map of This can be similar to "model-based" reinforcement learning. However, in the forward loop of the network, s t+1 The actual value is replaced by This eliminates the problem of error accumulation. The role of prediction is to propagate messages from the future back to past actions. In this sense, the algorithm can be a combination of "model-based" reinforcement learning and "policy-based learning".

[0243] An important element that can be provided in some scenarios is a differentiable path from future loss / reward back to the decision on the action. With the option graph structure, the implementation of options involving safety constraints is usually not differentiable. To overcome this problem, the selection of children in the learned policy node can be random. That is, the node can output a probability vector p that assigns the probability of each child for selecting a particular node. Suppose the node has k children, and let a (1) ,…,a (k) is the action of the path from each child to the leaf. The resulting predicted action is thus This may lead to a differentiable path from that action to p. In practice, action a can be chosen to be a for i~p (i) , and a and The difference between can be called additive noise.

[0244] For a given s t ,a t Down For training, supervised learning can be used with real data. For training of node policies, a simulator can be used. Afterwards, fine-tuning of the policy can be done using real data. Two concepts can make simulation more realistic. First, using imitation, an initial policy can be built using a large real-world dataset using the "behavioral cloning" paradigm. In some cases, the resulting actor can be suitable. In other cases, the resulting actor forms at least a very good initial policy for other actors on the road. Second, using self-play, our own policy can be used to reinforce training. For example, given an initial implementation of other actors (cars / pedestrians) that can be experienced, the policy can be trained based on a simulator. Some other actors can be replaced by new policies, and the process can be repeated. Therefore, the policy can continue to improve because it should respond to a wider variety of other actors with different levels of complexity.

[0245] Furthermore, in some embodiments, the system may implement a multi-actor approach. For example, the system may consider data from various sources and / or images captured from multiple angles. Furthermore, some disclosed embodiments may provide energy economy because anticipation of events that do not directly involve the host vehicle but may have an impact on the host vehicle may be considered, or even anticipation of events that may lead to unpredictable situations involving other vehicles may be a consideration (e.g., radar may "see through" the vehicle ahead and unavoidably anticipate, or even have a high probability of an event that will affect the host vehicle).

[0246] Global and local accuracy

[0247] In the context of autonomous driving, loss functions can be defined in order to fully define the accuracy of measurements from cameras, sensors, etc. (and thus impose conditions on them). Thus, a scene can be defined as a finite set S of objects (e.g., vehicles, pedestrians, lane markings, etc.). S contains the host vehicle, which is denoted as h. In this case, localization can include a mapping p:SS→R 3 , where p(h)=0=(0,0,0). Therefore, the first coordinate of p(a) may include the lateral position of object a, the second coordinate may include the longitudinal position of object a, and the last coordinate may include the height of object a.

[0248] Therefore, the loss function can be calculated with respect to two locations p and b of two objects a and b in the set S. The loss function can be defined as:

[0249]

[0250] It is usually impractical to impose constraints on the loss function. For example, if object a is a vehicle at position p(a) = (α, z, 0) and object b is a lane marker at position p(b) = (-α, z, 0), then the second positions of a and b are and Thus, if objects a and b are at a longitudinal position of 150 meters, and the loss β is 0.2 (ie, 20%), it is not possible to impose an absolute loss constraint of 20 centimeters, as this would make most losses unacceptable.

[0251] Therefore, the relative loss function can be defined as follows:

[0252]

[0253] where v is (0,1].

[0254] By normalizing the loss function, a realistic loss constraint can be imposed that takes into account larger losses for objects that are farther away. However, there are two ways to define accuracy using a normalized loss function. One is ego accuracy and is measured with respect to the host vehicle h:

[0255]

[0256] where ∈ is the loss constraint.

[0257] However, this requirement depends on the range of objects z detected in the vehicle's field of view, since if p(a)=(α,z,0) and

[0258]

[0259] To avoid dependence on the range z, an alternative precision definition can be used, which is pairwise:

[0260]

[0261] where ∈ is the loss constraint.

[0262] For p(a)=(α,z,0), p(b)=(-α,z,0), and The pairwise accuracy simplifies to:

[0263]

[0264] therefore,

[0265]

[0266] It is independent of z.

[0267] Furthermore, there are cases where there is self-precision but no pairwise precision. In particular, if but So that β=∈z. Therefore,

[0268]

[0269] This means that z>2 leads to a loss in pairwise accuracy.

[0270] In a specific example, if ∈ = 0.2 and z = 100 meters, then β = 2 meters, which is a reasonable loss constraint of 2 meters per 100 meters. However, this leads to This means that this situation deviates by a factor of 50 from 0.02-pairwise accuracy. To obtain pairwise accuracy, one can set z=1 meter and β=0.02, resulting in an error of 2 centimeters at a range of 100 meters. However, this is not realistic considering most existing sensors. On the other hand, pairwise accuracy can be realistically enhanced without considering ego-accuracy. To enhance this accuracy, a reference coordinate system can be used in which the relevant objects of the scene (e.g., vehicles, pedestrians, lanes, etc.) reside simultaneously. However, this may involve the use of cameras (rather than lidar, GPS, or other sensors).

[0271] Thus, in some embodiments, the host vehicle navigation system may use a two-dimensional (2-D) coordinate system of the camera, rather than a three-dimensional (3-D) coordinate system of the vehicle. The system may then convert a map (e.g., landmarks and splines of a sparse map) to a 2-D coordinate system and perform navigation in the 2-D coordinate system. Additionally, the system may perform navigation in a 3-D coordinate system by converting determinations made in the 2-D coordinate system to a 3-D coordinate system. This enforces pairwise accuracy, rather than ego accuracy, thereby providing greater safety and reliability. Additionally, this technique improves the efficiency of the system because converting a map to 2-D is faster than converting an image to 3-D, and because performing predictions and navigation in 2-D is faster than performing predictions and navigation in 3-D.

[0272] In one example embodiment, the navigation system can determine the location of the host vehicle. For example, the location can be within a geographic area. The navigation system can also access a map that includes the geographic area. For example, the navigation system can access a stored map, or access a map that includes the geographic area from one or more remote servers. In some embodiments, the map can include a sparse map or road guide (as described below) or a portion thereof based on the geographic area. As described above, the sparse map can include at least one spline representing a predetermined driving path and / or at least one landmark. Therefore, the at least one feature can include at least one landmark.

[0273] The navigation system may extract at least one feature from the map based on the position of the host vehicle. For example, the navigation system may determine a field of view of the host vehicle and extract at least one feature that is expected to be in the field of view based on the position. In some embodiments, the at least one feature may include a lane marking, a road edge, or other landmark included in the map. For example, the road edge may include at least one of a lane marking, a curb, a guardrail, or a Jersey wall.

[0274] The navigation system may receive at least one image representing the environment of the host vehicle from at least one image sensor, and convert the coordinates of at least one feature from the coordinate system of the map to the coordinate system of the at least one image sensor. For example, the coordinate system of the map may include a three-dimensional coordinate system (e.g., a global coordinate system based on GPS, a local coordinate system of a road segment included in a geographic area, etc.), and the coordinate system of the at least one image sensor may include a two-dimensional coordinate system based on the field of view of the at least one image sensor. In some embodiments, the at least one feature may be transformed from the three-dimensional coordinate system of the map to a three-dimensional coordinate system centered on the host vehicle (e.g., using the location), and then projected onto a two-dimensional plane of the at least one image sensor (e.g., using a known relationship between the host vehicle and the field of view).

[0275] The navigation system may analyze at least one image to identify at least one feature in the environment of the host vehicle and cause the host vehicle to make at least one navigation change based on a comparison between the transformed coordinates and the coordinates of the at least one feature identified in the at least one image. For example, the navigation system may determine an expected location of the at least one feature in a two-dimensional coordinate system of the at least one image sensor based on the transformed coordinates and search one or more images from the at least one image sensor at and / or near the expected location. The proximity may be determined absolutely, e.g., within 10 pixels, within 20 pixels, etc. of the expected location; or relatively, e.g., within 10% of an expected dimension (e.g., length or width) of the at least one feature, etc.

[0276] In some embodiments, at least one of the navigation changes may include decelerating the host vehicle, accelerating the host vehicle, or activating a steering mechanism of the host vehicle. For example, the host vehicle may decelerate, accelerate, and / or steer based on a difference between an identified position of the at least one feature in the at least one image and an expected position based on the transformed coordinates.

[0277] In some embodiments, at least one navigation change can be determined within the coordinate system of at least one image sensor. For example, a vector can be determined based on the difference between the identified position of at least one feature in at least one image and the expected position based on the transformed coordinates. The vector can represent at least one navigation change so that the at least one feature will appear where expected. In such an embodiment, the navigation system can convert the at least one navigation change to the coordinate system of the map. For example, the navigation system can project the difference vector into a three-dimensional coordinate system (e.g., a global coordinate system or a coordinate system centered on the host vehicle) based on the depth of the at least one feature in at least one image and / or the expected depth of the at least one feature from the map.

[0278] Fusion with comfort constraints and safety constraints

[0279] In some embodiments, the host vehicle may receive data from multiple sources, such as map data coupled with cameras, lidars, radars, etc. The navigation system of the host vehicle may use different schemes to fuse data from these various sources. For example, in a unification scheme, if at least one source detects a target object, the navigation system may verify the target object (i.e., confirm that a potential object should actually be considered a detected object), which is a fast but low-precision detection technique. In an intersection scheme, if multiple sources detect a target object, the navigation system may approve the target object; and in a synergy scheme, if the target object is detected using a combination of data from multiple sources, the navigation system may approve the target object. Intersection and synergy schemes are slower than unification schemes, but more accurate. Therefore, by selectively using intersection and synergy schemes and unification schemes, the accuracy of system reactions can be optimized without sacrificing safety. This solves the technical problem of how to accurately interpret sensor data from autonomous vehicles without sacrificing safety.

[0280] Fig.12 1 is an exemplary functional block diagram of memory 140 and / or memory 150, which may be stored / programmed with instructions for performing one or more operations consistent with the disclosed embodiments. Although the following refers to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or memory 150.

[0281] like Fig.12 As shown, the memory 140 may store an object recognition module 1202, a constraint module 1204, a verification module 1206, and a navigation change module 1208. The disclosed embodiments are not limited to any particular configuration of the memory 140. In addition, the application processor 180 and / or the image processor 190 may execute instructions stored in any of the modules 1202, 1204, 1206, and 1208 included in the memory 140. Those skilled in the art will appreciate that in the following discussion, references to the processing unit 110 may refer individually or collectively to the application processor 180 and the image processor 190. Therefore, the steps of any of the following processes may be performed by one or more processing devices.

[0282] In one embodiment, the object recognition module 1202 may store instructions (e.g., computer vision software) that, when executed by the processing unit 110, receive a first output from a first data source associated with the host vehicle and a second output from a second data source associated with the host vehicle. At least one of the first data source and the second data source includes a sensor onboard the host vehicle. For example, the object recognition module 1202 may receive a first output from a first sensor onboard the host vehicle and a second output from a second sensor onboard the host vehicle. Thus, the first data source may include at least one of a camera, a lidar, or a radar onboard the host vehicle, and the second data source may include at least one of a camera, a lidar, or a radar onboard the host vehicle that is different from the first data source.

[0283] Alternatively, the object recognition module 1202 may receive a first output from a first sensor onboard the host vehicle and a second output from a map accessed by the processing unit 110. Thus, the first data source may include at least one of a camera, lidar, or radar onboard the host vehicle, and the second data source may include map data.

[0284] In one embodiment, object recognition module 1202 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, recognize a representation of a target object in the first output. For example, object recognition module 1202 may perform all or part of process 500B described above to recognize a representation of a target object.

[0285] In one example, the object recognition module 1202 can determine a set of candidate objects representing a target object (e.g., a vehicle, a pedestrian, a stationary object, a lane marking, etc.) by scanning the first output, comparing the first output to one or more predetermined patterns, and identifying possible locations within the first output that may contain an object of interest (e.g., a vehicle, a pedestrian, a stationary object, a lane marking, etc.). The predetermined pattern can match the type of output from the first sensor. For example, if the first sensor is a camera, the predetermined pattern can be visual, and if the first sensor is a microphone, the predetermined pattern can be auditory. In some embodiments, the predetermined pattern can be configured to achieve a high "false hit" rate and a low "miss" rate. For example, the object recognition module 1202 can use a low similarity threshold with the predetermined pattern to identify the candidate object as a possible target object so as to reduce the likelihood of missing (e.g., not identifying) the candidate object representing the target object.

[0286] The object identification module 1202 may also filter the candidate object set to exclude certain candidate objects (e.g., irrelevant or less relevant objects) based on classification criteria. Such criteria may be derived from various attributes associated with object types stored in a database, such as a database stored in the memory 140 (not shown) and / or accessed from one or more remote servers. The attributes may include object shape, size, texture, location (e.g., relative to the host vehicle), speed (e.g., relative to the host vehicle), etc. Thus, the object identification module 1202 may use one or more sets of criteria to reject erroneous candidates in the candidate object set.

[0287] In an embodiment where the first output includes multiple frames over time, the object recognition module 1202 can also analyze multiple frames of the first output to determine whether an object in the candidate object set represents a target object. For example, the object recognition module 1202 can track detected candidate objects across consecutive frames and accumulate frame-by-frame data associated with the detected objects (e.g., size, position relative to the main vehicle, speed relative to the main vehicle, etc.). Additionally or alternatively, the object recognition module 1202 can estimate parameters of the detected objects and compare the frame-by-frame position data of the objects with the predicted position. The use of "frame" does not mean that the first output must be an image, although it can be an image. As used herein, "frame" refers to any discrete sequence of measurements across time received from a first sensor, a second sensor, or any additional sensor.

[0288] The object identification module 1202 may also construct a set of measurements for the detected objects. Such measurements may include, for example, position, velocity, and acceleration values ​​associated with the detected objects (e.g., relative to the host vehicle). In some embodiments, the target object module 2004 may construct these measurements based on an estimation technique using a series of time-based observations such as a Kalman filter or linear quadratic estimation (LQE), and / or based on available modeling data for different object types (e.g., cars, trucks, pedestrians, bicycles, road signs, etc.). The Kalman filter may be based on a measure of the size of the object, where the size measure is proportional to the collision time (e.g., the amount of time it takes the host vehicle to reach the object).

[0289] In an embodiment where the first output includes multiple frames over time, the object recognition module 1202 can perform optical flow analysis on one or more images to reduce the probability of detecting "false hits" and missing candidate objects representing vehicles or pedestrians. Optical flow analysis can refer to, for example, analyzing motion patterns related to the vehicle 200 in one or more images associated with other vehicles and pedestrians, and the motion patterns are distinguished from road motion. The processing unit 110 can calculate the motion of these candidate objects by observing the different positions of the candidate objects across multiple image frames (which are captured at different times). The processing unit 110 can use the position and time values ​​as inputs to a mathematical model for calculating the motion of the candidate objects. Therefore, optical flow analysis can provide another method for detecting vehicles and pedestrians near the vehicle 200. The processing unit 110 can perform optical flow analysis in conjunction with steps 540-546 to provide redundancy for detecting vehicles and pedestrians and increase the reliability of the system 100.

[0290] Additionally or alternatively, the object identification module 1202 may perform all or part of the process 500C described above to identify a representation of a target object. In an embodiment where the object identification module 1202 is implemented as an additional processing layer for a selected action of a trained system, the object identification module 1202 may receive an identification of a target object from the trained system. Accordingly, the object identification module 1202 may scan the first output; compare the first output with a pattern that matches the target object received from the trained system; and identify the location of the target object within the first output. For example, the object identification module 1202 may receive an identification of another vehicle from the trained network; extract a pattern that is stored in a database and indexed as a pattern of a vehicle and also matches a type (e.g., visual, auditory, thermal, etc.) of the first output, such as a database stored in the memory 140 and / or accessed from one or more remote servers (not shown); and identify the location of the other vehicle within the first output by comparing the first output with the extracted pattern.

[0291] Alternatively, in an embodiment where the object identification module 1202 is implemented as an additional processing layer for a selected action of a trained system, the object identification module 1202 may receive an identification of a target object and a location of the target object from the trained system. If the received location is in the first output, the object identification module 1202 may perform classification (e.g., using the comparison described above) at and / or near the received location to identify the target object in the first output. If the received location is in another output (e.g., from another sensor), the object identification module 1202 may: extract the following pattern, i.e., a pattern stored in a database and indexed as a pattern that matches the target object (e.g., vehicle, pedestrian, stationary object, etc.) and also matches the type of the first output (e.g., visual, auditory, thermal, etc.), the database is, for example, a database stored in the memory 140 and / or accessed from one or more remote servers (not shown); and compare the first output with the extracted pattern to identify the location of the target object within the first output. Additionally, or as an alternative to the comparison, the object identification module 1202 can construct an atlas that includes information mapping locations on the output used by the trained system to locations on the first output. Based on this, the object identification module 1202 can determine where a representation of the target object is expected to be in the first output based on the location of the target object in the output used by the trained system; and perform classification (e.g., using the comparison described above) to identify the location of the target object within the first output.

[0292] In one embodiment, the navigation constraint module 1204 may store software executable by the processing unit 110 to determine whether the characteristics of the target object trigger at least one navigation constraint. The characteristics of the target object may trigger hard (safety) constraints or soft (comfort) constraints. For example, the distance of the target object may trigger hard constraints based on minimum distance or maximum distance (e.g., distance to other vehicles, to pedestrians, to the edge of the road, to lane markings, etc.) and / or soft constraints based on preferred distances. In another example, the size of the target object may trigger hard constraints based on minimum size or maximum size (e.g., height of obstacles, height of gaps, etc.) and / or soft constraints based on preferred sizes. In yet another example, the location of the target object may trigger hard constraints based on restricted areas (e.g., within the current driving lane of the main vehicle, within a specific threshold distance of the expected trajectory of the main vehicle, etc.) and / or soft constraints based on preferred areas (e.g., on a lane or sidewalk near the current driving lane, within the range of the expected trajectory, etc.).

[0293] In one embodiment, if the characteristics of the target object do not trigger at least one navigation constraint, the verification module 1206 can verify the identity of the representation of the target object based on the combination of the first output and the second output. For example, the combination may include a cross scheme or a collaborative scheme. The cross scheme may include a requirement to identify the target object in both the first output and the second output for verification. For example, it may be necessary to identify the target object in a radar, a lidar, or a camera including a first data source and a radar, a lidar, or a camera including a second data source in order to verify the target object. That is, if the target object is detected by multiple data sources, it can be considered that the target object is approved. The collaborative scheme may include a combination of the first data source and the second data source to verify the target object. For example, the collaborative scheme may involve identifying or approving the target object based on combining partial data obtained from multiple data sources. An example of a collaborative scheme may include a camera estimate of the range of the target object, which is measured according to a road elevation model (e.g., based on another camera) or according to a lidar. Another example may include detection of the target object with a lidar and measurement of the target object using a road elevation model based on optical flow from one or more cameras. Another example may include detection of lanes using a camera (e.g., target objects include road edges or lane markings), and then confirming the detection using map data. Yet another example may include detection of target objects using one or more cameras, and using lidar to determine free space in the host vehicle's environment.

[0294] On the other hand, if the characteristics of the target object trigger at least one navigation constraint, the verification module 1206 can verify the identity of the representation of the target object based on the first output.For example, a unified scheme can be used so that only the first output is used to approve or verify the target object.

[0295] The navigation change module 1208 can use the output of the object recognition module 1202 and / or the verification module 1206 to implement a decision tree for navigation adjustments. The navigation adjustment can be based on data derived from the first sensor, the second sensor, any other sensor, the map data, and one or more objects detected from the first output, the second output, and any other output. The navigation change module 1208 can also determine the desired navigation response based on inputs from other systems of the vehicle 200, such as the throttle system 220, the braking system 230, and the steering system 240 of the vehicle 200. Additionally or alternatively, the navigation change module 1208 can receive one or more navigation adjustments from other memory modules (not shown) and / or from a trained system as described above. Therefore, the navigation change module 1208 can be implemented as an additional processing layer for selected actions of the trained system.

[0296] Therefore, in response to the above verification, the navigation change module 1208 can cause at least one navigation change of the host vehicle. To cause the at least one navigation change, the navigation change module 1208 can transmit electronic signals to the throttle system 220, the brake system 230, and the steering system 240 of the vehicle 200 to achieve a predetermined angle of rotation by, for example, turning the steering wheel of the vehicle 200 to trigger the desired navigation response. In some embodiments, the at least one navigation change can include any of the aforementioned adjustments to one or more navigation actuators of the host vehicle in response to the verification.

[0297] Furthermore, any module disclosed herein (e.g., module 1204, module 1204, module 1206, and module 1208) may implement techniques associated with a trained system (e.g., a neural network or a deep neural network) or an untrained system. Additionally or alternatively, any module disclosed herein (e.g., module 1204, module 1204, module 1206, and module 1208) may implement techniques as an additional processing layer for selected actions of a trained system.

[0298] Fig.13A and Fig. 13B A graphical description of example safety constraints and comfort constraints is provided. Fig.13A As shown, the main vehicle 1300 can detect other vehicles in front of the main vehicle 1300 (e.g., vehicle 1301), other vehicles behind the main vehicle 1300 (e.g., vehicle 1303), and other vehicles in lanes other than the lane in which the main vehicle 1300 is traveling (e.g., vehicle 1307) as target objects.

[0299] Such characteristics of detected objects may trigger navigation constraints. For example, distance 1309 between host vehicle 1300 and other vehicle 1301, distance 1311 between host vehicle 1300 and other vehicle 1307, and / or distance 1313 between host vehicle 1300 and other vehicle 1303 may trigger navigation constraints. Fig.13A and Fig. 13B Not shown, but other characteristics associated with one or more of vehicle 1301, vehicle 1303, and vehicle 1305 may include: relative speed between the main vehicle 1300 and one or more of vehicle 1301, vehicle 1303, and vehicle 1305, times-of-collision between the main vehicle and one or more of vehicle 1301, vehicle 1303, and vehicle 1305, etc.

[0300] In the above example, the triggered navigation constraints may be associated with: a distance to one or more of vehicle 1301, vehicle 1303, and vehicle 1305 (e.g., a minimum distance), a relative speed between host vehicle 1300 and one or more of vehicle 1301, vehicle 1303, and vehicle 1305 (e.g., a maximum relative speed, e.g., close to zero), a collision time with one or more of vehicle 1301, vehicle 1303, and vehicle 1305 (e.g., a minimum collision time, e.g., close to infinity), etc. Thus, these characteristics may trigger hard (or safety) constraints. Alternatively, these characteristics may not trigger navigation constraints. For example, one or more soft constraints (or "expectations") may be associated with these characteristics.

[0301] Fig. 13C and Fig.13D A graphical description of further example safety constraints and comfort constraints consistent with the disclosed embodiments is provided. Fig. 13C and Fig.13D As shown, the host vehicle 1300 can detect a stationary object 1315 on the road that the host vehicle 1300 is traveling on as a target object. Additionally or alternatively, the host vehicle 1300 can detect a car sign as a target object.

[0302] exist Fig. 13C and Fig.13D In the example of FIG. 1 , the distance 1317 between the host vehicle 1300 and the stationary object 1315 can be a characteristic that triggers the navigation constraint. Additionally or alternatively, the distance 1319 between the host vehicle 1300 and the lane marking can be a characteristic that triggers the navigation constraint. Fig. 13C and Fig.13D Not shown, but other characteristics associated with the stationary object 1315 or lane marking may include: the relative speed between the host vehicle 1300 and the stationary object 1315 or lane marking, the collision time between the host vehicle and the stationary object 1315 or lane marking, etc.

[0303] In the above examples, the navigation constraints can be associated with the distance to the stationary object 1315 or lane marking (e.g., a minimum distance), the relative speed between the host vehicle 1300 and the stationary object 1315 or lane marking (e.g., a maximum relative speed, such as approaching zero), the collision time with the stationary object 1315 or lane marking (e.g., a minimum collision time, such as approaching infinity), etc. Therefore, these characteristics can trigger hard (or safety) constraints. Alternatively, these features may not trigger navigation constraints. For example, one or more soft constraints (or "expectations") can be associated with these characteristics.

[0304] Fig.14A flow chart of an example process 1400 for navigating a host vehicle based on safety and comfort constraints consistent with the disclosed embodiments is provided. Process 1400 may be performed by at least one processing device, such as processing device 110 .

[0305] At step 1402, the processing device 110 may receive a first output from a first data source associated with the host vehicle. For example, as explained above with respect to the object recognition module 1202, the first data source may include at least one of a camera, a lidar, or a radar onboard the host vehicle.

[0306] At step 1404, the processing device 110 may receive a second output from a second data source associated with the host vehicle. For example, as explained above with respect to the object identification module 1202, the second data source may include map data accessed by at least one processing device.

[0307] In some embodiments, then, at least one of the first data source and the second data source comprises an onboard sensor on the host vehicle. In some embodiments, both the first data source and the second data source may comprise sensors. For example, the first data source and the second data source may comprise different cameras, the first data source may comprise a camera and the second data source may comprise a radar, the first data source may comprise a camera and the second data source may comprise a lidar, etc. In other embodiments, the other of the first data source and the second data source may comprise another data source, such as map data.

[0308] At step 1406, the processing device 110 may identify a representation of the target object in the first output. For example, as described above with respect to the object identification module 1202, the processing device 110 may identify the target object.

[0309] At step 1408, processing device 110 may determine whether a characteristic of the target object triggers at least one navigation constraint. For example, as explained above with respect to navigation constraint module 1204, the characteristic may include: the size of the target object, the distance from the host vehicle to the target object, or the location of the target object in the environment of the host vehicle.

[0310] At step 1410a, as explained above with respect to verification module 1206, if the characteristic of the target object does not trigger at least one navigation constraint, processing device 110 may verify the identity of the representation of the target object based on a combination of the first output and the second output.

[0311] In some embodiments, verifying the identity of the representation of the target object based on a combination of the first output and the second output may include determining whether the representation of the target object is recognized in both the first output and the second output. For example, as explained above with respect to the verification module 1206, the combination may include a crossover scheme.

[0312] Additionally or alternatively, verifying the identity of the representation of the target object based on the combination of the first output and the second output may include: using the second output projected onto the first output to determine the characteristics of the target object. For example, as explained above with respect to the verification module 1206, the combination may include a collaborative scheme. In one example, if the second output includes map data and the first output includes at least one image of the environment of the host vehicle, the above-mentioned projection may include detecting one or more road edges in the at least one image using the map data. In another example, if the second output includes an output from a lidar and the first output includes at least one image of the environment of the host vehicle, the above-mentioned projection may include detecting free space in the at least one image using the second output.

[0313] At step 1410b, as explained above with respect to the verification module 1206, if the characteristic of the target object triggers at least one navigation constraint, the processing device 110 may verify the identity of the representation of the target object based on the first output.

[0314] In response to the verification, the processing device 110 may cause at least one navigation change of the host vehicle at step 1412. For example, as explained above with respect to the navigation change module 1208, the at least one navigation change may include slowing down the host vehicle, accelerating the host vehicle, or activating a steering mechanism of the host vehicle.

[0315] Method 1400 may further include additional steps. For example, method 1400 may include determining at least one navigation change based on whether at least one navigation constraint is triggered. For example, as explained above with respect to navigation change module 1208, if at least one navigation constraint is triggered, at least one navigation change may include a first change; but if at least one navigation constraint is not triggered, at least one navigation change may include a different second change. In such an embodiment, the second change may include a narrower steering mechanism adjustment angle, a lighter braking mechanism application, a lighter acceleration, etc. than the first change.

[0316] Batch alignment for navigation

[0317] As explained above, a remote server can crowdsource sparse maps from multiple vehicles. However, global alignment of multiple drives leads to error accumulation in the sparse map. For example, ego-motion drift in a drive may distort the geometry of the road and may be exaggerated during global aggregation. In addition, global alignment using GPS data is usually inaccurate due to errors in GPS measurements.

[0318] Therefore, the remote server can locally align batch driving to generate road guides (roadbook), rather than globally align driving to generate sparse maps. As used herein, the term "road guide" may refer to a sparse map (as explained above) or other representations of position data (e.g., stored as one or more splines) and / or landmark data (e.g., stored as the location of a landmark and / or descriptive data related to the appearance, logo, etc. of a landmark) stored in the local coordinates of a road segment, rather than being stored in a global coordinate. The alignment of such data can result in a more reliable alignment and a smaller, more localized map. In addition, compared to aligning driving in global coordinates without prior local alignment, local road guides can be more accurately extrapolated to global coordinates. For example, self-drift can be considered during local alignment, so if local road guides are extrapolated to global coordinates, self-drift will not propagate. In addition, local alignment can be performed using visual cues such as lane markings, which can be more accurately positioned than using GPS data including inherent errors and drifts.

[0319] Furthermore, global alignment may not account for moving shadows, varying lighting, brightness due to rain, and other variations in images and data from multiple drives due to different times and dates the drives were performed. Therefore, batch alignment of drives taken on the same date, during similar times, and / or during similar weather conditions further improves the accuracy of road guidance.

[0320] Fig.15 1 is an exemplary functional block diagram of memory 140 and / or memory 150, which may be stored / programmed with instructions for performing one or more operations consistent with the disclosed embodiments. Although the following refers to memory 140, those skilled in the art will recognize that instructions may be stored in memory 140 and / or 150.

[0321] like Fig.15As shown, the memory 140 may store a navigation information receiving module 1502, an alignment module 1504, a storage module 1506, and a distribution module 1508. The disclosed embodiments are not limited to any particular configuration of the memory 140. In addition, the application processor 180 and / or the image processor 190 may execute instructions stored in any of the modules 1502, 1504, 1506, and 1508 included in the memory 140. Those skilled in the art will appreciate that in the following discussion, references to the processing unit 110 may refer to the application processor 180 and the image processor 190 individually or collectively. Alternatively, at least one processing device of a server remote from the host vehicle may execute instructions stored in any of the modules 1502, 1504, 1506, and 1508 included in the memory 140. Therefore, the steps of any of the following processes may be performed by one or more processing devices.

[0322] In one embodiment, the navigation information receiving module 1502 may store instructions (e.g., computer vision software) that, when executed by the processing unit 110, receive navigation information from multiple vehicles. For example, navigation information from multiple vehicles may be associated with a public road segment. The navigation information may be received via one or more computer networks. For example, multiple vehicles may upload information during driving, or multiple vehicles may upload information after completing driving, for example, during an upload session every hour, every day, every week, etc.

[0323] In some embodiments, the navigation information may include one or more images captured by one or more image sensors of the vehicle during driving. Additionally or alternatively, the navigation information may include processed information from the images, such as location and / or descriptive information about one or more landmarks identified in the images. The navigation information may additionally or alternatively include location information of the vehicle, such as GPS data.

[0324] In one embodiment, the alignment module 1504 may store instructions (e.g., computer vision software) that, when executed by the processing unit 110, align navigation information within a coordinate system local to the public road segment. For example, landmarks identified in an image from an image sensor of a vehicle may be used to align navigation information. In a simple scheme, alignment may include averaging the positions of landmarks detected in the image. In a more complex scheme, alignment may include linear regression or other statistical techniques to converge the positions of landmarks detected in the image. By using the image, the alignment module 1504 may, for example, based on GPS data, align navigation information in a local coordinate system rather than a global coordinate system. In practice, the alignment module 1504 may adjust the GPS data included in the navigation information based on the alignment of the landmarks, rather than adjusting the landmarks based on the alignment of the GPS data.

[0325] In one embodiment, the storage module 1506 may store instructions (e.g., computer vision software) that, when executed by the processing unit 110, store the aligned navigation information in association with the public road segment. For example, the navigation information may be stored in a database such that an identifier of the public road segment is stored with the aligned navigation information and the identifier of the public road segment is used to index the aligned navigation information. The identifier of the public road segment may include one or more coordinates for describing the public road segment (e.g., global coordinates of a start point of the public road segment and / or an end point of the public road segment).

[0326] In one embodiment, distribution module 1508 may store instructions (e.g., computer vision software) that, when executed by processing unit 110, distribute the aligned navigation information to one or more autonomous vehicles for autonomous navigation of the one or more autonomous vehicles along a public road segment. For example, one or more vehicles may request navigation information when the vehicle approaches or otherwise anticipates crossing a public road segment. Distribution module 1508 may transmit the aligned navigation information to the requesting vehicle via one or more computer networks.

[0327] Fig.16 Example road guides 1620 and 1640 generated by combining navigation information from many drives, and an example global map 1650 generated by combining road guides are shown consistent with the disclosed embodiments. Fig.16As shown, a first set of driving 1610 may include location data (e.g., GPS data) received from five independent driving along a public road segment. A driving may be independent of another driving in the following cases: if the driving is traversed by independent vehicles at the same time, or traversed by the same vehicle at different times, or traversed by independent vehicles at different times. The remote server may generate a road guide 1620 using one or more statistical techniques to align the location data along the road segment. For example, the remote server may determine whether a change in the location data represents an actual difference or a statistical error, and may align the location data using a coordinate system determined by images captured during the driving. Therefore, the alignment will be local to the road segment and will be self-consistent, rather than consistent with an external coordinate system (e.g., a global coordinate system).

[0328] Similarly, the second set of driving 1630 may include location data (e.g., GPS data) received from five additional driving along the common road segment. The remote server may use one or more statistical techniques to align the location data along the road segment to generate road guidance 1640. For example, the remote server may determine whether the changes in the location data represent actual differences or statistical errors, and may use a coordinate system determined by images captured during the driving to align the location data. Therefore, the alignment will be local to the road segment and will be self-consistent, rather than consistent with an external coordinate system (e.g., a global coordinate system).

[0329] The first set of drives 1610 and the second set of drives 1630 may be clustered by the remote server, for example, based on the time of day the drives were performed, the date the drives were performed, one or more weather conditions during the drives, etc. Thus, the road guides 1620 and the road guides 1640 may have increased accuracy compared to conventional techniques in which drives from different times of day, different dates, and / or different weather conditions are aligned with each other.

[0330] like Fig.16As further depicted in , the road guide 1620 and the road guide 1640 can be extrapolated to the global coordinate system and aligned as part of the global map 1650. For example, the remote server can again use one or more statistical techniques to align the location data along the road segment. The remote server can use GPS data or other data in the global coordinate system, rather than images captured during driving, to ensure that the alignment of the road guide 1620 and the road guide 1640 is performed in the global coordinate system rather than in the local coordinate system. Because the road guide 1620 and the road guide 1640 provide more accurate input than a single driver, the global map 1650 has a higher accuracy than assuming that the first group of drivers 1610 and the second group of drivers 1630 have been directly aligned on the global coordinate system.

[0331] Although described with driving data, road guide 1620 and road guide 1640 (and global map 1650) may also include one or more landmarks associated with road segments and present in the image. For example, landmarks may be aligned when forming road guide 1620 and road guide 1640 (or even landmarks may be used to align driving data forming road guide 1620 and road guide 1640). Similarly, landmarks may be aligned when forming global map 1650 (or even landmarks may be used to globally align road guide 1620 and road guide 1640).

[0332] Fig.17 A flow chart representing an example process 1700 for aligning navigation information from multiple vehicles consistent with the disclosed embodiments is provided. Process 1700 may be performed by at least one processing device, such as processing device 110. Alternatively, process 1700 may be performed by at least one processing device at a server remote from the host vehicle.

[0333] At step 1710, the server may receive navigation information from a plurality of vehicles. For example, the navigation information from the plurality of vehicles may be associated with a public road segment. In some embodiments, as explained above with respect to the navigation information receiving module 1502, the navigation information may include one or more landmarks contained in an image captured by a global positioning system (GPS) information and / or an image sensor included on the plurality of vehicles. For example, the one or more landmarks may include visible objects along the public road segment. In such embodiments, the object may include at least one of a road marking and a road sign.

[0334] In some embodiments, navigation information may be received via a computer network (e.g., cellular, Internet, etc.) using radio frequencies, infrared frequencies, magnetic fields, electric fields, etc. Any known standard for sending and / or receiving data (e.g., Wi-Fi, Bluetooth Smart, 802.15.4, ZigBee, etc.) to transmit navigation information.

[0335] At step 1720, the server may align the navigation information within a coordinate system local to the public road segment, as explained above with respect to the alignment module 1504. For example, the local coordinate system may include a coordinate system based on a plurality of images captured by image sensors included on a plurality of vehicles.

[0336] In some embodiments, the alignment of navigation information may be based on one or more landmarks. For example, as explained above, the server may adjust GPS data included in the navigation information based on the alignment of landmarks, rather than adjusting landmarks based on the alignment of GPS data.

[0337] At step 1730, the server may store the aligned navigation information in association with the public road segment as explained above with respect to the storage module 1506. For example, the navigation information may be stored in a database such that an identifier of the public road segment is stored with the aligned navigation information and is used to index the aligned navigation information.

[0338] At step 1740, as explained above with respect to distribution module 1508, the server can distribute the aligned navigation information to one or more autonomous vehicles for autonomous navigation of the one or more autonomous vehicles along the public road segment. For example, one or more vehicles can request navigation information, and the server can respond to the request and transmit the aligned navigation information over a computer network (e.g., cellular, Internet, etc.) using radio frequencies, infrared frequencies, magnetic fields, electric fields, etc. Any known standard for sending and / or receiving data (e.g., Wi-Fi, Bluetooth Smart, 802.15.4, ZigBee, etc.) to transmit aligned navigation information.

[0339] Method 1700 may also include additional steps. For example, in some embodiments, multiple vehicles may have captured navigation information during a particular time period. In such embodiments, method 1700 may also include: receiving additional navigation information from a second plurality of vehicles, the additional navigation information from the second plurality of vehicles being captured during the second time period and associated with the public road segment; and aligning the additional navigation information within a coordinate system local to the public road segment, the local coordinate system being a coordinate system based on multiple images captured by image sensors included on the second plurality of vehicles; and storing the aligned additional navigation information in association with the public road segment.

[0340] Additionally or alternatively, the plurality of vehicles may have captured navigation information during a number of drives that does not exceed a threshold number of drives. In such an embodiment, the method 1700 may also include receiving additional navigation information from a second plurality of vehicles, the additional navigation information from the second plurality of vehicles being captured during additional drives and associated with the public road segment; aligning the additional navigation information within a coordinate system local to the public road segment, the local coordinate system being a coordinate system based on a plurality of images captured by image sensors included on the second plurality of vehicles; and storing the aligned additional navigation information in association with the public road segment.

[0341] In any of the above embodiments, method 1700 may further include extrapolating the aligned navigation information to a global coordinate set, and storing the globally aligned navigation information in association with the public road segment. In addition, in embodiments including additional navigation information, method 1700 may further include: extrapolating the aligned navigation information and the aligned additional navigation information to a global coordinate set, and storing the globally aligned navigation information and the additional navigation information in association with the public road segment.

[0342] Align road information for navigation

[0343] As described above, the global alignment of sparse map data obtained from multiple driving can lead to error accumulation in the sparse map. The alignment techniques described herein can effectively reduce such errors. In some cases, such as areas where data points are relatively dispersed (e.g., in rural areas with fewer landmarks or other road features), additional processing techniques can be used. For example, the lower map data density in rural areas can involve aligning larger areas or aligning larger batches of map data. Errors caused by ego-motion drift inherent in vehicle data collection systems can be exacerbated at these longer distances, which can affect the accuracy of vehicle data alignment. Therefore, using the disclosed system and method, vehicle data can be aligned in multiple sections, wherein the data associated with each section can be aligned in a uniform manner with respect to other data within the section. In addition, each section can be fixed to each other at a common point and can rotate freely relative to each other around the common point. By aligning each section independently, errors caused by ego-motion drift or other inaccuracies can be reduced.

[0344] Fig.18AExample vehicle data that can be aligned consistent with the disclosed embodiments is shown. A first set of navigation information can be collected on a first drive 1810. The navigation information can include a series of points 1812 collected by a first vehicle 1831 navigating along a road segment. FIG. 18 also shows a second set of navigation information, which may have been collected by a second vehicle 1832 along a public road segment on a second vehicle drive 1820. Similar to the first navigation information, the second navigation information can include a series of points 1822 that have been collected by vehicle 1832. Each point can correspond to a road feature detected by vehicle 1831 and vehicle 1832 when traveling on a public road segment. For example, as described in more detail above, these points can represent road markings, road signs or traffic lights, poles, or other features along the road that can be used to generate a sparse map. As described above, vehicle 1831 and vehicle 1832 can correspond to vehicle 200. Therefore, any of the above descriptions or embodiments about vehicle 200 can also be applied to vehicle 1831 and / or vehicle 1832. Vehicle 1831 and vehicle 1832 may include a wireless transceiver, such as wireless transceiver 172, and may transmit navigation information to a server as described above.

[0345] The navigation information collected by vehicle 1831 may include point 1812A, and the navigation information collected by vehicle 1832 may include point 1822A. Point 1812A and point 1822A may correspond to the same detected feature. For example, point 1812A may represent a road sign or other road feature detected by vehicle 1831, and point 1822A may represent the same road feature detected by vehicle 1832 along driving 1820. Therefore, when driving 1810 is aligned with driving 1820, points 1812A and 1822A should ideally be located at the same location within the sparse graph.

[0346] Due to inherent errors associated with the vehicle collection systems, the data collected by vehicles 1831 and 1832 may "drift" as the vehicles travel along a road segment. For example, vehicles 1831 and 1832 may estimate their positions based on features in the images (also known as "ego-motion"). Small errors in estimating the vehicles' positions may accumulate over the length of the drive, and thus the data points may "drift" about the actual path traveled by the vehicles. As noted above, over longer distances this effect may be exacerbated. Additionally, road conditions (such as the curvature of the road) may increase the amount of ego-motion drift that may occur. In areas where many road features can be detected, such as in urban areas, the effects of ego-motion drift may be negligible because smaller batches of data can be easily aligned. In areas where the data is more sparse, such as in rural areas, larger batches of data may be used for alignment, and thus ego-motion drift may be more prominent. Examples of this effect are in Fig.18A , as shown in the shape of the data associated with drive 1810 and drive 1820 diverge over the length of the drive. Drive 1810 and drive 1820 may represent relatively long road segments where ego-motion drift may be a factor, such as road segments exceeding 100 meters, 300 meters, 500 meters, etc.

[0347] Fig.18B The error that may be introduced when aligning a greater distance of driving consistent with the disclosed embodiments is shown. Fig.18B , navigation information associated with driving 1810 and driving 1820 may be aligned. Thus, one or more data points about driving 1810 may be uniformly aligned with data points associated with driving 1820. For example, point 1812 may all be rigidly rotated and / or translated together to achieve optimal alignment with point 1822. Fig.18B As shown, points 1812 and 1822 may not be aligned with high accuracy. For example, while some of points 1812 and 1822 are aligned along the middle of driving 1810 and driving 1820, points 1812A and 1822A may be offset, indicating poor alignment between driving. This may lead to inaccuracies in the sparse map, which may translate into poor performance of a vehicle using the sparse map data for navigation. Although for illustrative purposes, in Fig.18A and Fig.18B Two sets of navigation road data are shown in FIG, but it should be understood that, consistent with the above-described embodiments, the disclosed method may be used to align many drives.

[0348] Fig.19A A plurality of segments 1910 that may be used to align road information are shown consistent with disclosed embodiments. Fig.19AAs shown, point 1810 can be divided into multiple segments 1910 (e.g., segment 1910A, segment 1910B, segment 1910C, etc.). These segments can be connected by multiple common points 1920. In some embodiments, as part of the alignment process, common points can be generated based on the length of segment 1910. Therefore, common point 1920 may not correspond to any data point acquired by the vehicle (e.g., point 1810 or 1820). In other embodiments, these segments can be determined so that common point 1920 is aligned with the data points collected by the vehicle. For example, segments 1910 can be determined so that they start and end on data points collected by the vehicle (such as point 1810 and point 1820). Consistent with the disclosed embodiments, navigation data can be divided into segments of any length suitable for increasing alignment accuracy. For example, the length of segment 1910 can be selected so as to minimize the impact of self-motion drift along the length of each segment 1910. In some embodiments, segment 1910 may be between 45 meters and 65 meters in length, although various other lengths may also be used.

[0349] The length of the segment 1910 can be determined based on various factors. In some embodiments, the length of the segment 1910 can be selected based at least in part on the data density along the driving 1810. For example, areas with relatively high data point density (e.g., more detectable road features along the road segment) can correspond to shorter segments, and vice versa. In some embodiments, the length of the segment 1910 can depend at least in part on the curvature of the road. For example, a road segment with a higher curvature can result in an increased error attributed to ego motion drift. Therefore, a shorter road segment 1910 can be used. In addition, in some embodiments, the length of the segment 1910 can depend at least in part on the amount of error associated with the driving 1810. For example, an error model can be used to determine the length of the segment 1810 to be used, which indicates the variability in how well the data is collected. The error model can be a general model, or it can be developed specifically for certain vehicle types, certain vehicle conditions, and / or certain road geometries or topologies. In some embodiments, each segment 1910 can have the same length. In other embodiments, segments 1910 may have different lengths, for example based on data density, road curvature, environmental conditions (e.g., lighting conditions, weather conditions, etc.), conditions of the host vehicle (e.g., tire pressure or other maintenance conditions), or other factors associated with points along the segment or the host vehicle. In some embodiments, the segment length may be selected so that the effect of ego-motion drift along one or more road segments is below a threshold. For example, ego-motion drift may be estimated based on the measured variance of position data compared to the positions of known landmarks. The segment length may be selected so that the measured variance is below a threshold variance. As described above, the estimated ego-motion drift or threshold may also be configured based on data density, road curvature, environmental conditions, conditions of the host vehicle, or other factors. Various other methods may also be used to determine the length of segment 1910 to improve the accuracy of alignment.

[0350] Each segment 1910 may be individually aligned with other driving information, such as with data points 1820. For example, segment 1910A may first be aligned with point 1820 corresponding to point 1810 included in segment 1910A. One or more points in segment 1910A may be rotated and / or translated together. Once segment 1910A has been aligned, segment 1910B may be aligned with corresponding point 1820. As with segment 1910A, data points within segment 1910B may be translated and / or rotated together to align with corresponding point 1820. Fig.19AAs shown, segment 1910A and segment 1910B can share a common point 1920A. When aligning the data points included in segment 1910B, the data points of segment 1910B can be rotated together about point 1920A, as shown by arrow 1912. The remaining segments, such as segment 1910C, can be aligned in a similar manner until the desired number of points 1810 are aligned. For example, the points can be aligned in a "chain" of multiple segments, where the points in each segment are translated and / or rotated together. Although the alignment process is described as occurring sequentially (e.g., segment 1910A, then segment 1910B, then segment 1910C, etc.), this is for illustrative purposes, and other orders of operations can be performed. For example, the alignment of the various segments can occur together, or can occur in any order. In some embodiments, alignment can be an iterative process. For example, one or more segments can be aligned first, and can be adjusted or fine-tuned to achieve a more precise alignment.

[0351] Thus, the effects of ego-motion drift or other errors associated with the navigation data may be minimized. Fig.19B The resulting alignment based on multiple sectors consistent with the disclosed embodiments is shown. In other words, the data of each sector 1910 can be aligned as described above. Fig.18B Compared to the alignment shown, more precise alignment can be achieved between point 1810 and point 1820 using multiple segments connected together in a "chain" through multiple common points. For example, point 1812A and point 1822A are Fig.19B can be compared to Fig.18B , without sacrificing the degree of alignment of other points 1810 and 1820. Therefore, the accuracy and efficiency of alignment for generating a sparse graph can be improved.

[0352] Fig. 20 2 is a flow chart representing an example process 2000 for aligning navigation information from multiple vehicles consistent with the disclosed embodiments. Process 2000 may be performed by at least one processing device, such as processing device 110. Alternatively, process 2000 may be performed by at least one processing device at a server remote from the host vehicle. In some embodiments, some or all of process 2000 may be performed using an alignment module, such as alignment module 1504 as described above.

[0353] At step 2010, process 2000 may include receiving first navigation information from a first vehicle. For example, as described above, the first navigation information may be received from vehicle 1831. The first navigation information may include a series of data points acquired by the first vehicle, such as point 1812. At step 2020, process 2000 may include receiving second navigation information from a second vehicle. For example, as described above, the second navigation information may be received from vehicle 1832. The second navigation information may include a series of data points acquired by the second vehicle, such as point 1822. For example, the first navigation information or the second navigation information includes global positioning system (GPS) information. The first navigation information and the second navigation information are associated with a public road segment, as described above with respect to Fig.18A In some embodiments, the first navigation information or the second navigation information is received via a network. For example, vehicle 1831 and vehicle 1832 may be configured to transmit the first navigation information and / or the second navigation information using a wireless transceiver as described above.

[0354] In some embodiments, as described above, navigation information may have been collected through multiple driving. For example, the first vehicle may have collected the first navigation information through a first number of driving, and the second vehicle may have collected the second navigation information through a second number of driving. In some embodiments, the first number of driving and the second number of driving may be equal. However, in other embodiments, the first number of driving and the second number of driving may not be equal. For example, the first vehicle may have collected the first navigation information through a greater number of driving than the second vehicle used to collect the second navigation information, and vice versa. In some embodiments, the first number of driving and the second number of driving may be equal to or less than a predetermined threshold number of driving. For example, a minimum number of driving may be required before the navigation information can be processed, which can ensure a higher accuracy of the information. In some embodiments, navigation information may be collected within the same time period. For example, the first vehicle may have collected the first navigation information during a predetermined time period, and the second vehicle may have collected the second navigation information during the predetermined time period.

[0355] At step 2030, process 2000 may include dividing the public road segment into at least a first road segment and a second road segment. For example, the public road segment may be divided into a first road segment 1910A and a second road segment 1910B, such as Fig.19AAs shown. The first road segment and the second road segment may be connected at a common point, such as at a common point 1920A. As described above, the lengths of the first road segment and the second road segment may be determined in various ways. In some embodiments, the length of the first road segment may be the same as the length of the second road segment. In other embodiments, the lengths of the first road segment and the second road segment may be different. In some embodiments, the length of at least one of the first road segment or the second road segment may be based at least in part on the curvature of at least a portion of the common road segment. Alternatively, or in addition, the length may be based on other factors, such as the density of navigation information, an error model associated with the navigation information, etc. In some embodiments, the length of the first road segment, the second road segment, or both may be between 45 meters and 65 meters.

[0356] At step 2040, process 2000 may include aligning the first navigation information with the second navigation information relative to a common road segment. For example, as described above, the road segments may be aligned based on the respective road segments. Aligning the first navigation information with the second navigation information may include rotating at least a portion of the first navigation information or at least a portion of the second navigation information relative to a common point, as described above with respect to Fig.19A For example, the first navigation information or the second navigation information can be rotated relative to the common point until at least a portion of the first navigation information at least partially overlaps at least a portion of the second navigation information. Fig.19B An example alignment according to process 2000 is shown.

[0357] like Fig.19A As shown, the public road segment can be divided into more than two road segments. For example, step 2030 can further include dividing the public road segment into a third road segment, such as road segment 1910C. The third road segment and the second road segment are connected at a second common point, such as common point 1920B. Therefore, aligning the first navigation information with the second navigation information can also include: rotating at least a portion of the first navigation information or at least a portion of the second navigation information relative to the second common point.

[0358] In some embodiments, the navigation information may include information based on identified landmarks. For example, the first navigation information may include a first position of at least one landmark based on one or more images captured by a camera included in the first vehicle. Similarly, the second navigation information may include a second position of at least one landmark based on one or more images captured by a camera included in the second vehicle. For example, the first vehicle and the second vehicle may capture images including representations of at least one landmark using the image acquisition unit 120. The processing device 110 may determine the position of at least one landmark, as described in more detail above. The at least one landmark may include, for example, a road sign, a road sign, or a traffic light. Aligning the first navigation information with the second navigation information may be based on the first position and the second position of the at least one landmark. For example, as described above, the first position and the second position of the at least one landmark may correspond to points 1812A and 1822A, where points 1812A and 1822A may be used to align driving information.

[0359] At step 2050, process 2000 may include storing the aligned navigation information in association with the public road segment. For example, the aligned navigation information may be stored in a database such that an identifier of the public road segment is stored in association with the aligned navigation information, and the identifier of the public road segment is used to index the aligned navigation information. In some embodiments, as described above, a map generated using the navigation information may include a three-dimensional coordinate system (e.g., a global coordinate system based on GPS, etc.). Therefore, step 2050 may also include associating the aligned navigation information with a global coordinate system and storing the associated navigation information in association with the public road segment.

[0360] At step 2060, process 2000 may include transmitting at least a portion of the aligned navigation information to one or more vehicles for use in navigating the one or more vehicles along the public road segment. In some embodiments, the one or more vehicles include an autonomous vehicle that may use the aligned navigation information to navigate along the public road segment.

[0361] The foregoing description has been presented for purposes of illustration. It is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will be apparent to those skilled in the art in view of the specification and practice of the disclosed embodiments. In addition, although aspects of the disclosed embodiments are described as being stored in a memory, those skilled in the art will appreciate that these aspects may also be stored on other types of computer-readable media, such as secondary storage devices, e.g., hard disks or CDROMs, or other forms of RAM or ROM, USB media, DVDs, Blu-rays, 4K Ultra HD Blu-rays, or other optical drive media.

[0362] Computer programs based on the written description and disclosed methods are within the skill of experienced developers. Various programs or program modules may be created using any technology known to those skilled in the art or may be designed in conjunction with existing software. For example, program parts or program modules may be designed with or through .NET Framework, .NET Compact Framework (and related languages, such as Visual Basic, C, etc.), JAVA, C++, Objective-C, HTML, HTML / AJAX combinations, XML, or HTML containing Java applets.

[0363] In addition, although illustrative embodiments have been described herein, those skilled in the art will recognize the scope of any and all embodiments with equivalent elements, modifications, omissions, combinations (e.g., throughout aspects of the various embodiments), adaptations and / or variations based on this disclosure. The limitations in the claims will be interpreted broadly based on the language adopted in the claims and are not limited to the examples described in this specification or during the examination of this application. The examples will be interpreted as non-exclusive. In addition, the steps of the disclosed methods may be modified in any way, including by reordering steps and / or inserting or deleting steps. Therefore, it is intended that the description and examples be regarded as illustrative only, and the true scope and spirit are represented by the full scope of the following claims and their equivalents.

Claims

1. A computer-implemented method for aligning navigation information from a plurality of vehicles, the method comprising: receiving first navigation information from a first vehicle; receiving second navigation information from a second vehicle, wherein the first navigation information and the second navigation information are associated with a road segment; dividing the first navigation information into at least a first part and a second part, and dividing the second navigation information into at least a first part and a second part, wherein the first part of the first navigation information and the first part of the second navigation information represent a first common section of the road segment, and wherein the second part of the first navigation information and the second part of the second navigation information represent a second common section of the road segment; aligning a first portion of the first navigation information with a first portion of the second navigation information, and aligning a second portion of the first navigation information with a second portion of the second navigation information; generating a road model based on the aligned sections; as well as transmitting at least a portion of the road model to one or more vehicles for use in navigating the one or more vehicles along the road segment; wherein the first navigation information includes a first position of at least one landmark based on one or more images captured by a camera included in the first vehicle, and the second navigation information includes a second position of the at least one landmark based on one or more images captured by a camera included in the second vehicle. 2 . The method of claim 1 , wherein the length of the first common section is the same as the length of the second common section. 3 . The method according to claim 1 , wherein the length of the first common section and the length of the second common section are both between 45 meters and 65 meters. 4 . The method of claim 1 , wherein a length of at least one of the first common section or the second common section is based at least in part on a curvature of at least a portion of the road segment.

5. The method according to claim 1, further comprising: dividing the first navigation information into a third portion, and dividing the second navigation information into a third portion, wherein the third portion of the first navigation information and the third portion of the second navigation information represent a third common portion of the road segment; as well as The third portion of the first navigation information is aligned with the third portion of the second navigation information. The method according to claim 1 , wherein the first navigation information or the second navigation information is received through a network.

7. The method of claim 1, wherein the first navigation information or the second navigation information comprises Global Positioning System (GPS) information. 8 . The method of claim 1 , wherein aligning the first portion of the first navigation information with the first portion of the second navigation information is based on the first and second positions of the at least one landmark.

9. The method of claim 1, wherein the at least one landmark comprises a road marking, a road sign, or a traffic light.

10. The method according to claim 1, further comprising: Associating the aligned portions with a global coordinate system; as well as The associated aligned portions are stored in association with the public road segment. 11 . The method of claim 1 , wherein the first vehicle collects the first navigation information and the second vehicle collects the second navigation information during a predetermined time period. 12 . The method of claim 1 , wherein the first vehicle collects the first navigation information through a first number of drives, and the second vehicle collects the second navigation information through a second number of drives.

13. The method of claim 12, wherein the first number of driving and the second number of driving are equal; the first number of driving and the second number of driving are not equal; or the first number of driving and the second number of driving are equal to or less than a predetermined threshold number of driving.

14. The method of claim 1, wherein the first navigation information or the second navigation information is rotated relative to a common point until at least a portion of the first navigation information at least partially overlaps at least a portion of the second navigation information.

15. The method of claim 1, wherein the one or more vehicles include an autonomous vehicle.

16. The method of claim 1, wherein aligning the first portion of the first navigation information with the first portion of the second navigation information comprises: At least one of the first portion of the first navigation information or the first portion of the second navigation information is rotated and translated.

17. The method of claim 1, wherein the first portion of the first navigation information and the second portion of the first navigation information are connected at a common point.

18. The method of claim 1, wherein the first portion of the second navigation information and the second portion of the second navigation information are connected at a common point.

19. The method of claim 1, wherein the first portion and the second portion of the second navigation information are associated with a first plurality of captured image frames, and wherein the first portion and the second portion of the second navigation information are associated with a second plurality of captured image frames.

20. A server for aligning navigation information from a plurality of vehicles, the server comprising: At least one processor programmed to perform a method according to any one of the preceding claims.

21. A non-transitory computer readable medium storing instructions which, when executed by at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 19.

Images