Keyframe-based compression for world model representation in autonomous systems and applications

Abstract

This disclosure relates to keyframe-based compression for world model representations in autonomous systems and applications. In various examples, a method includes computing a current keyframe, the current keyframe representing an area around an autonomous vehicle at a current time based on map data. The method includes transforming a previous keyframe to a coordinate system of the autonomous vehicle at a first time before completion of the computation of the current keyframe to generate a first world model frame. The method includes transforming the previous keyframe to the coordinate system of the autonomous vehicle at a second time after the first time and before completion of the computation of the current keyframe to generate a second world model frame.

Description

Technical Field

[0001] This disclosure generally relates to world models for autonomous vehicles, and more specifically to the compression of keyframe-based world model data. Background Technology

[0002] Autonomous and semi-autonomous vehicles, also known as self-driving cars, driverless cars, advanced driver-assistance vehicles, or robotic cars, are typically equipped with features that enable them to navigate from a source location to a destination location without the need for continuous monitoring, control, and / or direct operation of the vehicle by a human driver. Driving automation is difficult for several reasons. For example, autonomous vehicles use sensors to make driving decisions in real time, but vehicle sensors cannot always see everything. Vehicle sensors may be obstructed or blocked by corners, rolling hills, other vehicles, glare, objects, or road users (pedestrians, animals, cyclists, etc.). Input from vehicle sensors may not be processed early enough or quickly enough for autonomous planning or control functions to make appropriate decisions. Furthermore, lane markings and signs may be missing from the road or knocked down or hidden by bushes, making them undetectable by sensors. Additionally, road signs indicating right-of-way may not be easily seen to determine where a vehicle might be coming from or to indicate swerving or moving out of a lane in emergency situations or when there is a stopping obstacle that must be crossed.

[0003] Instead of relying solely on sensor data, autonomous vehicles can use accurate map data to alleviate some of the processing required to determine or verify some of the aforementioned information in real time. For example, autonomous vehicles can use map data to construct a world model aligned with their own coordinate system. For safe navigation, world model frames are typically calculated at a rate of approximately 30 world model frames per second. The large data volume of world model frames consumes significant communication, processing, and / or storage resources on the autonomous vehicle. Summary of the Invention

[0004] Embodiments of this disclosure relate to keyframe-based compression for world model (WM) systems and applications.

[0005] In an example embodiment, a method includes calculating a current keyframe representing the area surrounding an autonomous vehicle at the current time based on map data. The method further includes transforming a previous keyframe to the autonomous vehicle's coordinate system at a first time before the calculation of the current keyframe is completed, to generate a first WM frame. The method also includes transforming the previous keyframe to the autonomous vehicle's coordinate system at a second time after the first time and before the calculation of the current keyframe is completed, to generate a second WM frame.

[0006] In another example embodiment, a method includes calculating multiple keyframes representing one or more areas surrounding an autonomous vehicle at a first frequency based on map data. The method further includes transforming the most recent keyframe to the autonomous vehicle's coordinate system at a second frequency greater than the first frequency to generate frames, such that multiple WM frames are generated from at least one keyframe.

[0007] In another example embodiment, a system includes one or more processors to implement first and second WM buffers, a WM producer node, and a WM transformer node. The WM producer node is coupled to each of the first and second WM buffers to compute keyframes representing one or more areas surrounding the autonomous vehicle at a first frequency based on map data, and alternately populates the first and second WM buffers with these keyframes. The WM transformer node is coupled to each of the first and second WM buffers to transform the most recent keyframe stored in either the first or second WM buffer to the autonomous vehicle's coordinate system over time at a second frequency greater than the first frequency. Attached Figure Description

[0008] The system and method for WM keyframe-based compression of the present invention are described in detail below with reference to the accompanying drawings, wherein:

[0009] Figure 1 The illustration shows an example overall system environment of an HD map system capable of interacting with multiple vehicles according to one or more embodiments of the present disclosure;

[0010] Figure 2 The illustrations depict one or more embodiments of the present disclosure that may include... Figure 1 An example system architecture of a vehicle computing system in a system environment;

[0011] Figure 3 The illustrations depict one or more embodiments according to the present disclosure. Figure 1 Examples of various instruction layers in the HD map API of the vehicle computing system;

[0012] Figure 4 The illustrations depict one or more embodiments of the present disclosure that may include... Figure 1 Example system architecture of an online HD map system in a system environment;

[0013] Figure 5 Example components of an HD map according to one or more embodiments of the present disclosure are shown;

[0014] Figures 6A-6B Example geographic areas that can be defined in an HD map according to one or more embodiments of this disclosure are shown;

[0015] Figure 7 The illustration shows an example lane representation in an HD map according to one or more embodiments of the present disclosure;

[0016] Figure 8A and Figure 8B Example lane elements and relationships between lane elements in an HD map according to one or more embodiments of the present disclosure are shown;

[0017] Figure 9A An example architecture for computing WM frames according to one or more embodiments of the present disclosure is shown;

[0018] Figure 9B This illustrates one or more embodiments of the invention that can be derived from the present disclosure. Figure 9A The flowchart shows the methods implemented by the lane collector, segment graph generator, and / or other components in the architecture;

[0019] Figure 9C The illustration shows a method for generating WM frames according to one or more embodiments of the present disclosure;

[0020] Figure 10 An example lane graph calculation from a simple HD map is shown according to one or more embodiments of the present disclosure;

[0021] Figure 11A and Figure 11B An example lane map calculation from FIG11 for a moving autonomous vehicle or equipment is shown according to one or more embodiments of the present disclosure;

[0022] Figure 12 The illustration shows one or more embodiments of the present disclosure that may include [the following]. Figure 9A An example implementation of a three-node WM generator in the architecture;

[0023] Figure 13 This is a flowchart illustrating a method for generating WM frames according to one or more embodiments of the present disclosure;

[0024] Figure 14 This is a flowchart illustrating another method for generating WM frames according to one or more embodiments of the present disclosure;

[0025] Figure 15A These are illustrations of example autonomous vehicles according to some embodiments of the present disclosure;

[0026] Figure 15B According to some embodiments of this disclosure Figure 15A Examples of camera positions and fields of view for autonomous vehicles;

[0027] Figure 15C According to some embodiments of this disclosure Figure 15A A block diagram of an example system architecture for an example autonomous vehicle;

[0028] Figure 15D This is according to some embodiments of the present disclosure for use in one or more cloud-based servers and Figure 15A Example system diagram of communication between autonomous vehicles; and

[0029] Figure 16 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0030] Systems and methods related to processing large-scale map data in autonomous vehicles are disclosed. In one or more embodiments, the map data may include lanes, traffic signs and lights, waiting conditions, etc., and can be used at the autonomous vehicle to build a world model around the vehicle. Generally, the static content of each map-based world model is relatively slow-changing and therefore does not need to be generated for every frame. Therefore, some embodiments divide the data generation of the autonomous vehicle into multiple tasks, where tasks with higher latency are performed less frequently than tasks with lower latency. For example, data generation may be divided into a high-latency, low-frequency world model keyframe generation task and a low-latency, high-frequency transformation task. Alternatively or additionally, the tasks may include a low-latency, high-frequency transformation calculation task. These tasks may be repeated or iterated over time.

[0031] In an example embodiment, world model keyframes around the autonomous vehicle are computed over time and at a first frequency based on map data (e.g., lane maps). The latest world model keyframe at any given time is transformed into the autonomous vehicle's coordinate system over time at a second frequency greater than the first frequency to generate multiple world model frames, thereby generating multiple world model frames from each world model keyframe. In another example embodiment, the current world model keyframe around the autonomous vehicle is computed based on map data. A previous world model keyframe can be transformed into the autonomous vehicle's coordinate system at a first time before the computation of the current world model keyframe is completed to generate a first world model frame. A previous world model keyframe can also be transformed into the autonomous vehicle's coordinate system at a second time after the first time and before the computation of the current world model keyframe is completed to generate a second world model frame. In both example embodiments, the higher-latency computation of world model keyframes is performed at a lower frequency than the lower-latency generation of world model frames.

[0032] Some embodiments may implement dual buffers for world model keyframes. For example, when computing world model keyframes, world model keyframes may be alternately filled in a first world model buffer or a second world model buffer. While computing the current world model keyframe and filling the current world model keyframe in the first or second world model buffer, a previously or most recently computed world model keyframe in the other of the second or first world model buffer is repeatedly transformed to the autonomous vehicle's coordinate system to generate a world model frame. The compression ratio in some embodiments herein may depend on the speed of the autonomous vehicle. For example, when the autonomous vehicle is traveling at approximately 90 mph or approximately 40 m / s, the compression ratio may be at least 10:1 (e.g., 10:1, 12:1, 20:1, or any other ratio where the first number is at least ten times the second number). Alternatively or additionally, when the autonomous vehicle is traveling at approximately 22 mph or approximately 10 m / s, the compression ratio may be at least 40:1 (e.g., 40:1, 45:1, 50:1, or any other ratio where the first number is at least forty times the second number).

[0033] Figure 1 The illustration depicts an exemplary overall system environment of a high-definition (HD) map system 100 capable of interacting with multiple vehicles according to one or more embodiments of the present disclosure. The HD map system 100 may include an online HD map system 110 capable of interacting with two or more vehicles 150 (e.g., vehicles 150A-150D) of the HD map system 100. Vehicles 150 may be autonomous vehicles, semi-autonomous vehicles, or non-autonomous vehicles. Exemplary embodiments of vehicle 150 are described below. Figures 15A-15D The following are illustrations and descriptions. It should be understood that the arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groups, etc.) may be used in addition to or in lieu of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory.

[0034] Continue to refer to Figure 1The online high-definition map system 110 can be configured to receive sensor data that can be captured by vehicle sensors 105 (e.g., 105A-105D) of vehicle 150, and combine the data received from vehicle 150 to generate and maintain a high-definition map. The online high-definition map system 110 can be configured to send high-definition map data to vehicle 150 for driving vehicle 150. In some embodiments, the online high-definition map system 110 can be implemented as a distributed computing system, such as a cloud-based service, that allows clients such as vehicle computing systems 120 (e.g., vehicle computing systems 120A-120D) to request information and services. For example, vehicle computing system 120 can request high-definition map data for driving along a route, and the online high-definition map system 110 can provide the requested high-definition map data to vehicle computing system 120. Vehicle computing system 120 can perform world model keyframe-based compression on the high-definition map data received from the online high-definition map system 110.

[0035] Figure 1 The same reference numerals are used to identify the same elements as in other figures. A letter following a reference numeral, such as "105A", indicates that the text specifically refers to the element having that particular reference numeral. A reference numeral without a following letter in the text, such as "105", refers to any or all elements in the figure that have that reference numeral (e.g., "105" in the text refers to reference numerals "105A" and / or "105N" in the figure).

[0036] The online high-definition map system 110 may include a vehicle interface module 160 and a high-definition map repository 165. The online high-definition map system 110 can be configured to interact with a vehicle computing system 120 of various vehicles 150 using the vehicle interface module 160. The online high-definition map system 110 can be configured to store map information for various geographical areas in the high-definition map repository 165. The online high-definition map system 110 can be configured to include, in addition to... Figure 1 Other modules besides the ones shown, for example, Figure 4 The various other modules shown and further described herein.

[0037] In this disclosure, a module may include code and routines configured to enable a corresponding system (e.g., a corresponding computing system) to perform one or more of the operations described therewith. Additionally or alternatively, any given module may be implemented using hardware including any number of processors, microprocessors (e.g., those performing one or more operations or controlling the execution of one or more operations), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or any suitable combination of two or more thereof. Alternatively or additionally, any given module may be implemented using a combination of hardware and software. In this disclosure, the operations described as being performed by the module may include operations that the module may instruct the corresponding system to perform.

[0038] Furthermore, the distinction and separation of different modules indicated in this disclosure are for the purpose of aiding in the explanation of the operations being performed and are not intended to be limiting. For example, according to an embodiment, two or more described operations relating to the modules described in this disclosure may be performed by modules that can be considered the same. Additionally, depending on the embodiment, the operations of one or more modules may be divided among other modules or sub-modules that can be considered one or more other modules.

[0039] The online HD map system 110 can be configured to receive sensor data collected by sensors from two or more vehicles 150 (e.g., hundreds or thousands of vehicles). The sensor data may include any data that can be obtained by the vehicle's sensors and may be relevant to the generation of the high-definition map. For example, the sensor data may include LiDAR data, captured images, etc. Alternatively, the sensor data may include information that can describe the current state of the vehicle 150, the vehicle 150's position, and motion parameters, etc.

[0040] Vehicle 150 can be configured to provide sensor data 115, which can be captured while traveling along various routes, and transmit it to an online high-definition map system 110. The online high-definition map system 110 can be configured to use the sensor data 115 received from vehicle 150 to create and update HD maps describing areas where vehicle 150 may be traveling. The online high-definition map system 110 can be configured to construct high-definition maps based on the collective sensor data 115 that can be received from vehicle 150 and store the high-definition map information in a high-definition map repository 165. One method for generating high-definition maps has been described above. More generally, the HD maps described herein can be generated using the foregoing and / or other suitable methods.

[0041] The online HD map system 110 can be configured to send HD map data 125 to vehicle 150 upon request.

[0042] For example, when a specific vehicle 150 is scheduled to travel along a route, the specific vehicle calculation system 120 for that specific vehicle 150 can be configured to provide information describing the route being traveled to an online high-definition map system 110. In response, the online high-definition map system 110 can be configured to provide high-definition map data 125 of a high-definition map associated with the route (e.g., indicating an area including the route), which can facilitate navigation and travel by the specific vehicle 150 along the route.

[0043] In one embodiment, the online high-definition map system 110 can be configured to send portions of high-definition map data to vehicle 150 in a compressed format, thereby reducing the bandwidth consumption of the transmitted data. The online high-definition map system 110 can be configured to receive from various vehicles 150 descriptions of a local high-definition map repository that can be stored within the vehicle 150 (e.g., Figure 2 Information on high-definition map data from the local high-definition map repository (275).

[0044] In some embodiments, the online high-definition map system 110 may determine that a particular vehicle 150 may not have locally stored certain portions of its high-definition map data in the local high-definition map repository of the particular vehicle's computing system 120. In these or other embodiments, in response to such a determination, the online high-definition map system 110 may be configured to send the specific portions of the high-definition map data to the vehicle 150.

[0045] In some embodiments, the online high-definition map system 110 may determine that a particular vehicle 150 may have previously received high-definition map data about the same geographic area as a specific portion of the high-definition map data. In these or other embodiments, the online high-definition map system 110 may determine that the specific portion of the high-definition map data may be an updated version of previously received high-definition map data that has been updated by the online high-definition map system 110 since the particular vehicle 150 last received previous high-definition map data. In some embodiments, the online high-definition map system 110 may send updates to the portion of the high-definition map data that can be stored at the particular vehicle 150. This may allow the online high-definition map system 110 to reduce or minimize the number of high-definition maps that can communicate with the vehicle 150, and is also used to periodically update the high-definition map data stored locally in the vehicle.

[0046] Vehicle 150 may include vehicle sensors 105 (e.g., vehicle sensors 105A-D), vehicle controllers 130 (e.g., vehicle controllers 130A-130D), and vehicle computing systems 120 (e.g., vehicle computer systems 120A-120D). Vehicle sensors 105 may be configured to detect the surrounding environment of vehicle 150. In these or other embodiments, vehicle sensors 105 may detect information describing the current state of vehicle 150, such as information describing the position and motion parameters of vehicle 150.

[0047] Vehicle sensor 105 may include a camera, a light detection and ranging sensor (LIDAR), a Global Navigation Satellite System (GNSS) receiver such as a Global Positioning System (GPS) navigation system, an inertial measurement unit (IMU), etc. Vehicle sensor 105 may include one or more cameras that can capture images of the vehicle's surroundings. The LIDAR can survey the vehicle's surroundings by measuring the distance to a target by illuminating it with a laser pulse and measuring the reflected pulse. The GPS navigation system can determine the vehicle 150's position based on signals from satellites. The IMU may include electronics configured to measure and report motion data of the vehicle 150, such as rate, acceleration, direction of motion, velocity, angular rate, etc., using a combination of accelerometers and gyroscopes or other measuring instruments.

[0048] The vehicle controller 130 can be configured to control the physical movement of the vehicle 150, such as acceleration, direction change, starting, stopping, etc. The vehicle controller 130 may include mechanical devices for controlling accelerators, brakes, steering wheels, etc. The vehicle computing system 120 can periodically and / or continuously provide control signals to the vehicle controller 130 and can cause the vehicle 150 to travel along a selected route.

[0049] The vehicle computing system 120 can be configured to perform various tasks, including processing data collected by sensors and map data received from the online high-definition map system 110. The vehicle computing system 120 can also be configured to process data intended for transmission to the online high-definition map system 110. An example of the vehicle computing system 120 is provided in... Figure 2 The text further illustrates and combines... Figure 2 Further description.

[0050] Interaction between the vehicle computing system 120 and the online high-definition map system 110 can be performed via a network, such as the Internet. The network can be configured to enable communication between the vehicle computing system 120 and the online high-definition map system 110. In some embodiments, the network can be configured to utilize standard communication technologies and / or protocols. Data exchanged over the network can be represented using technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. Furthermore, conventional encryption techniques can be used to encrypt all or part of the links, such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. In some embodiments, entities can use custom and / or dedicated data communication technologies.

[0051] Vehicle computing system

[0052] Figure 2 An example system architecture of a vehicle computing system 120 according to one or more embodiments of the present disclosure is illustrated. The vehicle computing system 120 may include a perception module 210, a prediction module 215, a planning module 220, a control module 225, a local high-definition map repository 275, a high-definition map system interface 280, a map difference module 290, and a high-definition map application programming interface (API) 205. The various modules of the vehicle computing system 120 can be configured to process various types of data, including sensor data 230, behavioral models 235, routes 240, and physical constraints 245. In some embodiments, the vehicle computing system 120 may contain more or fewer modules. Functions described as being implemented by a particular module may be implemented by other modules.

[0053] refer to Figure 2 and Figure 1 In some embodiments, the vehicle computing system 120 may include a perception module 210. The perception module 210 may be configured to receive sensor data 230 from vehicle sensors 105 of the vehicle 150. The sensor data 230 may include data collected by cameras from the vehicle, LiDAR, IMU, GPS navigation system, etc. The perception module 210 may also be configured to use the sensor data 230 to determine what objects are around the corresponding vehicle 150, details of the road the corresponding vehicle 150 is traveling on, etc. Furthermore, the perception module 210 may be configured to process the sensor data 230 to populate a data structure storing the sensor data 230 and to provide information or instructions to the prediction module 215 of the vehicle computing system 120.

[0054] The prediction module 215 can be configured to interpret the data provided by the perception module 210 using the behavior model (235) of the perceived object to determine whether the object is moving or is likely to move. For example, the prediction module 215 can determine that an object representing a road sign is unlikely to move, while objects identified as vehicles, people, etc., are likely to be in motion or likely to move. The prediction module 215 can also be configured to use the behavior models 235 of various types of objects to determine whether they are likely to move. Furthermore, the prediction module 215 can also be configured to provide predictions of various objects to the planning module 200 of the vehicle computing system 120 to plan the subsequent actions that the corresponding vehicle 150 may take next.

[0055] The planning module 220 can be configured to receive information from the prediction module 215 describing the surrounding environment of the corresponding vehicle 150 and a route 240, which indicates or determines the destination of the vehicle 150 and can indicate the path that the vehicle 150 needs to take to reach the destination.

[0056] The planning module 220 can also be configured to use information from the prediction module 215 and the route 240 to plan a series of actions that the vehicle 150 may take within a short time interval (e.g., in the next few seconds). In some embodiments, the planning module 220 can be configured to designate a series of actions as one or more points representing nearby locations that the vehicle 150 may subsequently travel through. The planning module 220 can be configured to provide the control module 225 with details of the planned series of actions to be taken by the corresponding vehicle 150. This plan may indicate one or more subsequent actions of the corresponding vehicle 150, such as whether the corresponding vehicle 150 can perform lane changes, turns, acceleration, etc., by increasing or decreasing speed.

[0057] Control module 225 can be configured to determine control signals that can be sent to vehicle controller 130 of the corresponding vehicle 150 based on planning information received from planning module 220. For example, if the corresponding vehicle 150 is currently at point A and planning specifies that the corresponding vehicle 150 should next proceed to a nearby point B, control module 225 can determine control signals for vehicle controller 130 that could cause the corresponding vehicle 150 to proceed from point A to point B in a safe and smooth manner, without requiring any sharp turns or zigzag paths from point A to point B. The path that the corresponding vehicle 150 can take from point A to point B can depend on the current speed and direction of the corresponding vehicle 150 and the position of point B relative to point A. For example, if the current speed of the corresponding vehicle 150 is high, the corresponding vehicle 150 may take wider turns compared to another slower-moving vehicle.

[0058] The control module 225 can also be configured to receive physical constraints 245 as input. Physical constraints 245 may include the physical capabilities of the corresponding vehicle 150. For example, a corresponding vehicle 150 with a specific brand and model may be able to safely perform certain types of vehicle movements, such as acceleration and cornering, while another vehicle with a different brand and model may not be able to safely perform the same acceleration and cornering. Furthermore, the control module 225 can be configured to incorporate physical constraints 245 when determining control signals for the vehicle controller 130 for the corresponding vehicle 150. Additionally, the control module 225 can be configured to send control signals to the vehicle controller 130 that cause the vehicle 150 to execute a specified sequence of actions and to cause the corresponding vehicle 150 to move according to a predetermined set of actions. In some embodiments, the aforementioned steps may be repeated continuously every few seconds, and may cause the corresponding vehicle 150 to travel safely along a route that may have been planned for the corresponding vehicle 150.

[0059] Various modules of the vehicle computing system 120, including perception module 210, prediction module 215, and planning module 220, can be configured to receive map information to perform their respective calculations. The corresponding vehicle 150 can store high-definition map data in a local high-definition map repository 275. The modules of the vehicle computing system 120 can interact with the high-definition map data using a high-definition map application programming interface (API) 205.

[0060] HD Map API 205 can provide one or more application programming interfaces (APIs) that can be used for module calls to access map information. HD Map System Interface 280 can be configured to allow Vehicle Computing System 120 to interact with Online HD Map System 110 via a network (not shown). Local HD Map Repository 275 can store map data in a format that can be specified by Online HD Map System 110. HD Map API 205 can be configured to process map data formats provided by Online HD Map System 110. HD Map API 205 can be configured to provide Vehicle Computing System 120 with an interface for interacting with HD map data. HD Map API 205 may include several APIs, including Location API 250, Landmark Map API 255, Route API 270, 3D Map API 265, Map Update API 285, etc.

[0061] The positioning API 250 can be configured to determine the current location of the corresponding vehicle 150, such as the location of the corresponding vehicle 150 relative to a given route. The positioning API 250 can be configured to include a localized API for determining the location of the corresponding vehicle 150 within a high-resolution map and at a specific accuracy level. The vehicle calculation system 120 can also be configured to use this location as an accurate (e.g., within a certain accuracy level) relative location for other queries, such as feature queries, navigable space queries, and occupancy map queries, as further described herein.

[0062] The positioning API 250 can be configured to receive inputs including one or more of the following: location provided by GPS, vehicle motion data provided by IMU, LiDAR scanner data, camera images, etc. The positioning API 250 can be configured to return the accurate location of the corresponding vehicle 150 as latitude and longitude coordinates. Compared to GPS coordinates used as input, the coordinates returned by the positioning API 250 may be more accurate; for example, the output of the positioning API 250 may have an accuracy in the range of 2-30 cm. In some embodiments, the vehicle computing system 120 can be configured to invoke the positioning API 250 to periodically determine the location of the corresponding vehicle 150 based on LiDAR scanner data, for example at a frequency of 10 Hz.

[0063] The vehicle computing system 120 can also be configured to invoke the positioning API 250 to determine the vehicle's position at a higher rate (e.g., 60Hz) when GPS or IMU data is available at that rate. Furthermore, the vehicle computing system 120 can be configured to store location history as internal state to improve the accuracy of subsequent positioning calls. The location history can store location history starting from a point in time, such as when the corresponding vehicle 150 is turned off / stopped. The positioning API 250 can include a positioning route API, which can be configured to generate an accurate (e.g., within a specified accuracy) route for a specified lane based on a high-definition map. The positioning route API can be configured to receive routes from source to destination as input via one or more third-party maps and can be configured to generate a high-precision (e.g., within a specified accuracy, such as within 30cm) route based on the high-definition map, represented as a connection map of navigable lanes along the input route.

[0064] Landmark Map API 255 can be configured to provide a geometric and semantic description of the world surrounding the corresponding vehicle 150, such as a description of the various parts of the lane the corresponding vehicle 150 is currently traveling in. Landmark Map API 255 includes APIs that can be configured to allow queries based on the landmark map, such as a Lane Get API and a Feature Get API. The Lane Get API can be configured to provide lane information relative to the corresponding vehicle 150 and the Feature Get API. The Lane Get API can also be configured to receive a location, such as the location of the corresponding vehicle 150 specified using latitude and longitude, as input and return lane information relative to the input location. Furthermore, the Lane Get API can be configured to specify a distance parameter indicating the distance relative to the input location for which lane information can be retrieved. Additionally, the Feature Get API can be configured to receive information identifying one or more lane elements and return landmark features relative to the specified lane elements. For each landmark, the landmark features can include a landmark-type-specific spatial description.

[0065] The 3D Map API 265 can be configured to provide access to a spatial 3D representation of roads and various physical objects around them, such as those stored in a local HD map repository 275. The 3D Map API 265 may include a fetch-navigable-surfaces API and a fetch-occupancy-grid API. The fetch-navigable-surfaces API can be configured to receive identifiers of one or more lane elements as input and return the navigable boundaries of the specified lane elements. The fetch-occupancy-grid API can also be configured to receive a location as input, such as the latitude and longitude corresponding to vehicle 150, and return information describing the occupancy of all objects available in the HD map near the road surface and location. The occupancy information may include a layered volumetric grid of some or all locations considered occupied in the HD map. The occupancy grid may include high-resolution information near navigable areas, such as at curbs and bumps, and relatively lower resolution in less important areas, such as trees and walls outside the curb. Furthermore, the fetch-occupancy-grid API can be configured to be useful for obstacle detection and direction changes if desired.

[0066] The 3D map API 265 also includes map update APIs, such as a download map update API and an upload map update API. The download map update API can be configured to receive a planned route identifier as input and download map updates related to all planned routes or a specific planned route. The upload map update API can be configured to upload data collected by the vehicle computing system 120 to the online high-definition map system 110. The upload map update API allows the online high-definition map system 110 to keep the high-definition map data stored in the high-definition map system 110 up-to-date based on changes in the map data, changes that can be observed by the vehicle sensors 105 of the vehicles 150 traveling along various routes.

[0067] Route API 270 can be configured to return route information as the corresponding vehicle 150 travels along the route. Route information includes the complete route between the source and destination, as well as portions of the route. 3D Map API 265 can be configured to allow querying of an online high-definition map system 110 or a high-definition map. Route API 270 can include an Add Route Planning API and a Get Route Planning API. The Add Route Planning API can be configured to provide information describing the planned route to the online HD map system 110, enabling the information describing the relevant HD map to be downloaded by the vehicle computing system 120 and kept up-to-date. The Add Route Planning API can be configured to receive a route specified using a polyline in latitude and longitude as input, and can also receive a Time-to-Live (TTL) parameter for a specified time period after which the route data can be deleted. Therefore, the Add Route Planning API can be configured to allow vehicle 150 to instruct itself to plan a route to be taken in the near future as an autonomous journey. The Add Route Planning API can be configured to align the route with the HD map, record the route and its TTL value, and ensure that the HD map data of the route stored in the vehicle computing system 120 is updated (e.g., up-to-date). The get-planned-routes API can be configured to return a list of planned routes and provide information describing the routes identified by route identifiers.

[0068] Map Update API 285 can be configured to manage operations related to updating map data for both the local high-definition map repository 275 and the high-definition map repository 165 stored in the online high-definition map system 110. Therefore, modules in the vehicle computing system 120 can be configured to call Map Update API 285 to download data from the online high-definition map system 110 to the vehicle computing system 120 for storage in the local high-definition map repository 275. Map Update API 285 can also be configured to allow the vehicle computing system 120 to determine whether information monitored by vehicle sensors 105 indicates discrepancies in the map information provided by the online high-definition map system 110, and to upload the data to the online high-definition map system 110, which may cause the online high-definition map system 110 to update the map data stored in the high-definition map repository 165 provided to other vehicles 150.

[0069] Map discrepancy module 290 can be configured to operate in conjunction with map update API 285 to determine map discrepancies and transmit map discrepancy information to online HD map system 110. In some aspects, determining map discrepancies involves comparing sensor data 230 at a specific location with high-definition map data for that location. For example, high-definition map data may indicate that a highway lane should be available for vehicle 150, but sensor data 230 may indicate that construction work occurring on that lane has been closed or is no longer available for other reasons. When map discrepancy module 290 detects a map discrepancy, the corresponding vehicle 150 sends an update message to online HD map system 110, which includes information about the detected map discrepancy. Map discrepancy module 290 can be configured to construct an update message that may include a vehicle identifier (ID), one or more timestamps, the route traveled, lane element IDs of lane elements traversed, type of discrepancy, size of discrepancy, a discrepancy fingerprint to help identify duplicate discrepancy alert messages, message size, etc. In some embodiments, one or more operations of map discrepancy module 290 may be at least partially described in detail below. Figure 4 The map data collection module 460 processes the data.

[0070] In some embodiments, the corresponding vehicle 150 may be configured to send an update message to the online high-definition map system 110 or to the local high-definition map repository 275 and / or periodically send update messages when a map difference is detected. For example, the corresponding vehicle 150 may be configured to record the difference and report it to the online high-definition map system 110 via an update message once at each time interval (e.g., 30 minutes) or once at each distance traveled (e.g., 10 miles). The online high-definition map system 110 may be configured to manage and prioritize update messages, as described in more detail below with reference to the map data collection module 460.

[0071] In some embodiments, the corresponding vehicle 150 may be configured to send update messages to the online high-definition map system 110 only upon arrival at or docking with a high-bandwidth access point. Once the corresponding vehicle 150 is connected to the Internet (e.g., a network), it may be configured to send a consolidated update message or a set of update messages. In one or more embodiments, example non-limiting messages may include update messages built up since arrival at or docking with the previous high-bandwidth access point. High-bandwidth access points can be used to transmit large amounts of data. In some aspects, upon receiving an acknowledgment message from the online high-definition map system 110 confirming receipt of a consolidated update message or one or more update messages, the corresponding vehicle 150 marks data for deletion to schedule a local deletion process and / or delete the data. Alternatively, the corresponding vehicle 150 may report to the online high-definition map system 110 periodically (e.g., hourly) based on time.

[0072] Map difference module 290 can be configured to function in response to messages from online high-definition map system 110 and perform operations related to difference identification. For example, upon receiving a message requesting data on a specific location along the route of corresponding vehicle 150, map difference module 290 can be configured to instruct one or more vehicle sensors 105 of corresponding vehicle 150 to collect the data and report it to map difference module 290. After receiving the data, map difference module 290 can be configured to construct a message containing the data and send the message to online high-definition map system 110 immediately, at the next predetermined time on a periodic schedule, or at the next high-bandwidth access point, etc.

[0073] Map discrepancy module 290 can be configured to determine the urgency of identified map discrepancies to be included in updates to any HD map containing areas with discrepancies. For example, there may be two levels of urgency: low urgency and high urgency. Online HD map system 110 can consider the urgency of update messages when determining how to process the information in the update message, as detailed below regarding map data collection module 460. For example, a single-lane closure on a remote or rural road might be determined to have low urgency, while the complete closure of a major highway in a city with a million inhabitants might be determined to have high urgency. In some cases, high-urgency update messages can be processed by online HD map system 110 before low-urgency update messages.

[0074] In some embodiments, the corresponding vehicle 150 may be configured to continuously record sensor data 230 and encode relevant portions thereof for use in generating messages to the online high-definition mapping system 110, such as in response to a request for additional data for a specific location. In one embodiment, the vehicle 150 may be configured to delete the continuously recorded sensor data 230 only when it is confirmed from the online high-definition mapping system 110 that the online high-definition mapping system 110 does not require the sensor data 230.

[0075] Figure 3 Examples of various instruction layers in a high-definition map API 205 of a vehicle computing system 120 according to one or more embodiments of this disclosure are illustrated. Different vehicle manufacturers may have different programs or instructions for receiving information from vehicle sensors 105 and for controlling the vehicle controller 130. Furthermore, different vendors may provide different computer platforms with autonomous driving capabilities, such as for the collection and analysis of vehicle sensor data. Examples of computer platforms for autonomous vehicles include vendor-supplied platforms such as NVIDIA, Qualcomm, and Intel. These platforms can provide functionality for autonomous vehicle manufacturers to use in manufacturing the autonomous vehicle 150. Vehicle manufacturers may use any one or more of these computer platforms for the autonomous vehicle 150.

[0076] The online high-definition map system 110 can be configured to provide a library for processing high-definition maps based on vehicle manufacturer-specific instructions and vehicle-specific supplier-specific platform instructions. This library provides access to high-definition map data and allows vehicle 150 to interact with the online high-definition map system 110.

[0077] like Figure 3 As shown, the high-definition map API 205 can be implemented as a library including a vehicle manufacturer adapter 310, a computer platform adapter 320, and a universal high-definition map API layer 330. The universal high-definition map API layer 330 can be implemented to include general instructions that can be used across two or more vehicle computing platforms and vehicle manufacturers. The computer platform adapter 320 can be implemented to include instructions specific to each computer platform. For example, the universal high-definition map API layer 330 can be implemented to invoke the computer platform adapter 320 to receive data from sensors supported by a specific computer platform. The vehicle manufacturer adapter 310 can be implemented to include vehicle manufacturer-specific instructions. For example, the universal high-definition map API layer 330 can be implemented to invoke functions provided by the vehicle manufacturer adapter 310 to send specific control commands to the vehicle controller 130.

[0078] The online high-definition map system 110 can be implemented to store computer platform adapters 320 for two or more computer platforms and vehicle manufacturer adapters 310 for two or more vehicle manufacturers. The online high-definition map system 110 can be implemented to determine a specific vehicle manufacturer and a specific computer platform for a specific autonomous vehicle 150. The online high-definition map system 110 can be implemented to select a vehicle manufacturer adapter 310 for a specific vehicle manufacturer and a computer platform adapter 320 for the specific computer platform of that specific vehicle 150. Furthermore, the online high-definition map system 110 can be implemented to send instructions from the selected vehicle manufacturer adapter 310 and the selected computer platform adapter 320 to the vehicle computing system 120 of that specific autonomous vehicle 150. The vehicle computing system 120 of that specific autonomous vehicle 150 can be implemented to install the received vehicle manufacturer adapter 310 and computing platform adapter 320. The vehicle computing system 120 can be implemented to periodically check or verify whether the online high-definition map system 110 has updates for the installed vehicle manufacturer adapter 310 and computing platform adapter 320. Additionally, if a newer update is available compared to the version installed on vehicle 150, vehicle computing system 120 can be configured to request and receive the latest update and install it.

[0079] High-definition map system architecture

[0080] Figure 4 The illustration shows an example system architecture of an online high-definition map system 110 according to one or more embodiments of the present disclosure. The online high-definition map system 110 may include a map creation module 410, a map update module 420, a map data encoding module 430, a load balancing module 440, a map accuracy management module 450, a vehicle interface module 160, a map data collection module 460, and an HD map repository 165. Some embodiments of the online HD map system 110 may include... Figure 4 The number of modules may be more or less. Functions indicated as being performed by a particular module can be implemented by other modules. In some embodiments, the online high-definition map system 110 may be implemented as a distributed system comprising two or more processing systems.

[0081] Map creation module 410 can be configured to create high-definition map data based on sensor data collected from several vehicles (e.g., 150A-150D) traveling along various routes. Map update module 420 can be configured to update previously calculated HD map data by receiving more recent information (e.g., sensor data) from vehicles 150 that have recently traveled along routes where map information has changed. For example, some road signs may have changed or lane information may have changed due to construction in the area, and map update module 420 can be configured to update the HD map and corresponding HD map data accordingly. Map data encoding module 430 can be configured to encode the HD map data to enable efficient data storage (e.g., compressing the HD map data) and to send the HD map data to vehicles 150. Load balancing module 440 can be configured to balance the load among vehicles 150 such that requests for receiving data from vehicles 150 are distributed (e.g., evenly distributed) among the different vehicles 150 (e.g., the load among the different vehicles 150 is distributed within a threshold amount between them). The map accuracy management module 450 can be configured to use various technologies to maintain a relatively high level of accuracy for HD map data, even if the information received from each vehicle may not have the same level of accuracy.

[0082] In some embodiments, the map data collection module 460 may be configured to monitor vehicle 150 and process status updates from vehicle 150 to determine whether to request one or more specific vehicles 150 to obtain additional data related to one or more specific locations.

[0083] Figure 5 Example components of a high-resolution map 510 according to one or more embodiments of the present disclosure are illustrated. The high-resolution map 510 can be configured to include high-resolution map data of maps of several geographic regions. In this disclosure, references to a map or a high-resolution map such as high-resolution map 510 may include references to map data corresponding to such a map. Furthermore, references to information about various maps may also include references to map data for that map.

[0084] In some embodiments, the HD map 510 of a geographic area may include a landmark map (LMap) 520 and an occupancy map (OMap) 530. The landmark map 520 may include information or representations of driving routes (e.g., lanes, yield lines, safe navigation spaces, carriageways, unpaved roads, etc.), pedestrian paths (e.g., crosswalks, sidewalks, etc.), and landmark objects (e.g., road signs, buildings, etc.). For example, the landmark map 520 may include information describing lanes, including the spatial location of the lanes and semantic information about each lane. The spatial location of the lanes may include high-precision latitude, longitude, and elevation geometry, for example, with an accuracy of 30 cm or better. The semantic information of the lanes includes restrictions such as direction, speed, lane type (e.g., straight lane, left-turn lane, right-turn lane, exit lane, etc.), left-turn restrictions, and connections to other lanes.

[0085] In some embodiments, the landmark map 520 may include information describing the spatial locations of stop lines, yield lines, pedestrian crossings, safe navigation spaces, speed bumps, curbs, road signs, road signs including their spatial locations, and all types of signs related to driving restrictions. Examples of road signs described in the HD map 510 may include traffic signs, stop signs, traffic lights, speed limits, one-way signs, no entry signs, yield signs (for vehicles, pedestrians, and animals), etc.

[0086] In some embodiments, the information included in the landmark map 520 may be associated with a confidence value that measures the probability of an accurate representation. An object's representation is considered accurate when the information describing the object matches the object's attributes (e.g., driving path, pedestrian path, landmark object, etc.). For example, a driving path's representation can be considered accurate when the spatial location and semantic information of the driving path match the attributes of the driving path (e.g., physical measurements, restrictions, etc.). The vehicle calculation system 120 (e.g., planning module 220) can use the confidence value to control the vehicle 150. For example, if a landmark object's representation is associated with a high confidence value in the landmark map 520, but the vehicle 150 does not detect the landmark object based on the vehicle sensors 105 and corresponding observations of the environment surrounding the vehicle 150, the vehicle calculation system 120 can be configured to control the vehicle 150 to avoid the hypothetical landmark object based on the high confidence value, or to control the vehicle 150 to follow driving restrictions imposed by the landmark object (e.g., causing the vehicle 150 to yield based on yield signs on the landmark map).

[0087] In some embodiments, the occupancy map 530 may include a spatial 3D representation of roads and physical objects around the roads. The occupancy map 530 may also be referred to herein as an occupancy grid. Similarly, the data stored in the occupancy map 530 may also be referred to herein as occupancy grid data. The 3D representation of roads and physical objects around the roads may be associated with a confidence score indicating the probability that an object is present at that location. The occupancy map 530 may be represented in a variety of other ways. In some embodiments, the occupancy map 530 may be represented as a 3D mesh geometry (a set of triangles) that can cover a surface. In some embodiments, the occupancy map 530 may be represented as a set of 3D points that can cover a surface. In some embodiments, the occupancy map 530 may be represented using a 3D volumetric mesh with cells having a resolution of 5-10 cm. Each cell may indicate whether a surface is present at that cell, and if so, indicate the direction in which the surface can be oriented.

[0088] Compared to Landmark Map 520, Occupation Map 530 may require significantly more storage space. For example, Occupation Map 530 could use 1GB / mile of data to generate a map of the United States (including 4 million miles of roads), occupying 4x10... 15 Bytes or 4 petabytes (PB). Therefore, the online high-definition map system 110 and the vehicle computing system 120 can be configured to use data compression techniques to store and transmit map data, thereby reducing storage and transmission costs. Therefore, the techniques disclosed herein can help improve the autonomous driving of autonomous vehicles by increasing the efficiency of data storage and transmission of information regarding autonomous driving operations and capabilities.

[0089] In some embodiments, the HD map 510 may not use or rely on data typically included in a map, such as addresses, road names, the ability to geocode addresses, and the ability to calculate routes between place names or addresses. The vehicle computing system 120 or the online HD map system 110 may access other map systems, such as (but not limited to) Open Street Map, to obtain this information. Therefore, the vehicle computing system 120 or the online HD map system 110 may receive map information from tools such as Open Street Map and may convert that information into routes based on the HD map 510, or may convert the information to be compatible with the HD map 510.

[0090] Geographic regions in high-definition maps

[0091] The online HD map system 110 can be configured to divide a large physical area into geographic regions and store a representation of each geographic region. Each geographic region can represent a contiguous area bounded by a geometric shape—such as (but not limited to) a rectangle or a square. In some embodiments, the online HD map system 110 can be configured to divide the physical area into geographic regions of similar size, regardless of the amount of data required to store a representation of each geographic region. In some embodiments, the online HD map system 110 can divide the physical area into geographic regions of different sizes, wherein the size of each geographic region can be determined based on the amount of information required to represent that geographic region. For example, a geographic region representing a densely populated area with many streets may represent a smaller physical area than a geographic region representing a sparsely populated area with few streets. In some embodiments, the online HD map system 110 can be configured to determine the size of the geographic regions based on an estimate of the amount of information available for storing various elements of the physical area associated with the HD map 510.

[0092] In one embodiment, the online high-definition map system 110 may use objects or data records that may include various attributes, including (but not limited to): a unique identifier for the geographic region; a unique name for the geographic region; a description of the geographic region's boundaries, for example, using bounding boxes corresponding to a set of latitude and longitude coordinates; and a set of landmark features and occupancy grid data.

[0093] Figures 6A-6B Example geographic areas 610A and 610B that can be defined in a high-definition map according to one or more embodiments of the present disclosure are shown. Figure 6A The diagram illustrates a square geographic region 610A. Figure 6B The illustration shows two adjacent geographic regions 610A and 610B. The online high-definition map system 110 can be configured to store data in a representation of geographic regions, which allows for a smooth transition from one geographic region to another when the vehicle 150 travels across the boundaries of the geographic regions.

[0094] In some embodiments, such as Figures 6A-6B As shown, each geographic region may include a buffer of a predetermined width (area) surrounding it. The buffer may include redundant map data surrounding one or more or all sides of the geographic region (e.g., in the case where the geographic region is defined by a rectangle). Therefore, in some embodiments where the geographic region may be of a particular shape, the geographic region may be defined by a buffer that may be a larger version of that shape. For example, Figure 6A The boundary 620 of the buffer zone approximately 50 meters around geographic area 610A and the boundary 630 of the buffer zone approximately 100 meters around geographic area 610A are shown.

[0095] In some embodiments, the vehicle calculation system 120 can be configured to switch the current geographic region of the corresponding vehicle 150 from one geographic region to an adjacent geographic region when the corresponding vehicle 150 crosses a predetermined (e.g., defined) threshold distance within a buffer. For example, as Figure 6B As shown, vehicle 150 starts at position 650A within geographic region 610A. Vehicle 150 can traverse the route to reach position 650B, where it can cross the boundary of geographic region 610A but can remain within the boundary 620 of the buffer. Therefore, vehicle calculation system 120 of vehicle 150 can continue to use geographic region 610A as the current geographic region of vehicle 150. Once vehicle 150 crosses the boundary 620 of the buffer at position 650C, vehicle calculation system 120 can be configured to switch the current geographic region of vehicle 150 from geographic region 610A to geographic region 610B. Because vehicle 150 travels along a route that closely follows the boundaries of geographic regions, the use of the buffer can reduce or prevent rapid switching of the current geographic region of vehicle 150.

[0096] Lane indication in high-definition maps

[0097] HD map system 100 can represent lane information of streets in an HD map. Although the described embodiments may relate to streets, these techniques are applicable to any highway, alley, avenue, boulevard, path, etc., on which vehicles can travel. HD map system 100 can use lanes as a reference frame for route planning and positioning of vehicle 150. Lanes represented by HD map system 100 may include explicitly marked lanes, such as white and yellow striped lanes, implied lanes, such as on rural roads without lines or curbs but potentially with two directions of travel, and implied paths that may serve as lanes, such as the path taken by a turning vehicle when entering a lane from another lane.

[0098] HD map system 100 can store lane-related information, such as lane-related landmark features like road signs and traffic lights, lane-related occupancy grids for obstacle detection, and lane-related navigable space, so that vehicle 150 can plan / respond in emergency situations when it accidentally leaves its lane. Therefore, HD map system 100 can store a representation of the lane network to allow vehicle 150 to plan a legal path between source and destination and add a reference frame for real-time sensing and control of vehicle 150. HD map system 100 stores information and provides APIs that allow vehicle 150 to determine its current lane, its precise position relative to lane geometry, and any and all relevant features / data relative to the lane and adjacent and connected lanes.

[0099] Figure 7 The illustration shows an example lane representation in a high-definition map according to one or more embodiments of the present disclosure. Figure 7 A vehicle 710 is shown at a traffic intersection. The HD map system 100 provides the vehicle 710 with access to map data that may be relevant to the autonomous driving of the vehicle 710. This may include, for example, features 720A and 720B, which may be associated with a lane but may not be the features closest to the vehicle 710. Therefore, the HD map system 100 can store lane-centric data representations that can indicate the relationship between lanes and features, enabling the vehicle 710 to efficiently extract features from a given lane.

[0100] HD map system 100 can provide an HD map that represents portions of a lane as lane elements. Lane elements can specify lane boundaries and various constraints, including the legal direction a vehicle 710 can travel within the lane element, the speed at which a vehicle can travel within the lane element, and whether the lane element can only be for left or right turns, etc. In some embodiments, HD map system 100 can provide a map that represents lane elements as continuous geometric portions of a single vehicle lane. HD map system 100 can store objects or data structures that can represent lane elements, including information representing lane geometric boundaries; the direction of travel along the lane; vehicle restrictions within the lane, such as speed limits; relationships with connecting lanes, including entering and exiting lanes; termination restrictions, such as whether the lane terminates at a stop line, yield sign, or speed bump; and relationships with road features relevant to autonomous driving, such as traffic light locations, road sign locations, etc.

[0101] Examples of lane elements represented by the high-definition map of the high-definition map system 100 may include, for example, but not limited to: a right lane on a highway, a lane segment on a road, a left-turn lane, a lane merging from a left-turn lane into another lane, a merging lane from an entrance ramp, an exit lane on an exit ramp, and a driving lane. The high-definition map system 100 may include a high-definition map using two lane elements to represent a single-lane road, one for each direction. The HD map system 100 may represent a middle turning lane similar to those shared by single-lane roads.

[0102] Figure 8A and Figure 8B The relationship between example lane elements (e.g., LaneEl) and lane elements in an HD map is illustrated according to one or more embodiments of the present disclosure. Figure 8A and Figure 8B Each lane element may include one or more map lanes described elsewhere in this document, be included in one or more map lanes described elsewhere in this document, or correspond to one or more map lanes described elsewhere in this document. Figure 8A An example of a T-junction in a road is illustrated, showing lane element 810A (e.g., an example of a straight Lane El), which can be connected to lane element 810C (e.g., another straight Lane El) via turning lane 810B (e.g., a curved Lane El) and to lane 810E (e.g., another straight Lane El) via turning lane 810D (e.g., another curved Lane El). Figure 8B An example of a Y-shaped intersection in a road is shown, where lane 810F is directly connected to lane 810H and connected to lane 810I via lane 810G. The HD map system 100 can determine the route from the source location to the destination location as a sequence of connected lane elements that can be traversed to reach the destination location from the source location.

[0103] An example architecture and method for building a local map around an autonomous vehicle will now be described. In the following discussion: MapsLane refers to a lane defined by polylines with lane attributes; LaneSegment refers to a mutually exclusive segment formed by stitching together MapsLane; RoadSegment refers to an atomic element of the core layer of the high-resolution map; LocalLayout refers to a segment of the high-resolution map, where the relative transformations of all RoadSegments are calculated with reference to the root RoadSegment; LaneGraph refers to a detailed list of all path combinations in the LocalLayout; LaneGraph lanes refer to drivable paths in the LocalLayout; and LaneInfo refers to intermediate metadata of MapsLane used for LaneGraph calculation. The foregoing and other specific terms in this document can be generalized to refer to the corresponding general structures, concepts, etc. For example, "lane map" can refer to an exhaustive list of all path combinations in the driver of a high-definition map, where the relative transformations of all atomic elements in the core layer of the high-definition map are calculated with reference to the root atomic element, whether the segment of the high-definition map is called LocalLayout, and whether the atomic element is called RoadSegment.

[0104] LocalMap can provide a partial view of an HD map, which is achieved by calculating an exhaustive LaneGraph containing all path combinations in the LocalLayout. During the calculation of the LaneGraph, other high-definition map features from the LocalLayout may be generated, such as WaitGroups, TSL Objects, etc.

[0105] In one or more embodiments, WaitGroups may include a collection or grouping of individual waiting elements. In at least one embodiment, the waiting elements constitute a single or atomic representation of how a waiting condition (a potential yielding scenario) is represented. Each waiting element consists of a self-competitor pair (e.g., consisting of the vehicle itself and at least one competing obstacle or object in the environment) and a waiting state describing the current right-of-way of the waiting element. Wait groups (“WaitGroup”) associate multiple waiting elements together. In one or more embodiments, waiting elements are grouped together for waiting conditions intended to be cleared together. For example, driving on an unprotected left-hand side might be indicated as having a requirement to yield to pedestrians at a crosswalk at the end of a turn. Grouping these two waiting elements makes it clearer that the vehicle should wait to enter its own lane for any waiting element until both waiting elements can be cleared.

[0106] Traffic Sign (TSL) objects encode the 3D geometry of traffic lights / signs. In one or more embodiments, waiting rules can be used to define the association between waiting states and light categories. These associations are used to resolve the current waiting state applicable to a waiting element. For example, a typical waiting rule for a waiting element might look like "take-way" when green, "stop-at-entry" when red, and "take-way-translate" when yellow. In one or more embodiments, the TSL objects and waiting rules can be implemented using internal metadata and used when processing the resolution of the waiting state corresponding to a waiting element.

[0107] Each lane in the LaneGraph can know its other related lanes (merging, splitting, intersecting lanes), lane-changing lanes (left, right lanes), and other map features (WaitGroups, TSL objects, etc.). The LaneGraph and HD map features can be represented in the autonomous vehicle's coordinate system, which may be alternatively or additionally called the equipment coordinate system or ego coordinate system. The world model definition can be used as the output of the LocalMap to express this data.

[0108] World model calculation

[0109] Figure 9AAn example architecture 900 for computing World Model (WM) frames according to one or more embodiments of this disclosure is illustrated. As shown, architecture 900 may include a lane collector 902, a segmented graph generator 904, a lane graph generator 906, and a WM generator 908. Typically, lane collector 902 collects and maintains MapsLane in LocalLayout, segmented graph generator 904 stitches MapsLane to mutually exclusive LaneSegments to generate a coarse map (or SegmentGraph), lane graph generator 906 combines all lane combinations from LaneSegments, where each lane may have its own unique tracking identifier (ID), which may be consistent across frames, and WM generator 908 populates WM structures, such as WM keyframes, for lanes, waiting conditions, etc., and can transform the computed lane graph to its own coordinate system and determine its own lanes. The output of WM generator 908 may be referred to as a WM frame. Each of the lane collector 902, segment map generator 904, lane map generator 906, and WM generator 908 will be discussed in more detail below.

[0110] Lane collector 902 can abstract a collection of lanes from LocalLayout. Each lane in LocalLayout can be stored in a wrapper structure. In the example embodiment, the wrapper structure for each lane in LocalLayout is called a LocalLayoutLane.

[0111] Lane collector 902 may have a lane buffer, which is filled by the current LocalLayout.

[0112] Lane collector 902 can collect lanes by iterating through all road segments in the current LocalLayout construction.

[0113] A LocalLayout for LaneGraph computation can be constructed around a predetermined radius surrounding a sensor suite (also known as "equipment") typically arranged along the perimeter of an autonomous vehicle. The predetermined radius around the equipment can define a circular region, with the equipment located at the center of the circular region. The circular region can be referred to as the LocalLayout and / or the boundary or perimeter of the circular region can be referred to as the horizontal line of the autonomous vehicle or equipment.

[0114] The segmented graph generator 904 can construct a coarse map composed of local map segments. MapsLane in a LocalLayout can be stitched together to form mutually exclusive LocalMapSegments, where at any given time a MapsLane will be associated with only one LocalMapSegment. A LocalMapSegment may know its other related LocalMapSegments (entry, exit, intersection, and lane-changing segments). A LocalMapSegment can be an atomic unit of the coarse map, also referred to here as a segment graph.

[0115] MapsLanes participating in a LocalMapSegment can be encoded in a MapsLane structure. The MapsLane structure may include the ID of the participating MapsLane and / or its offset within the LocalMapSegment (this information can be used to infer lane changes and other lane assignment attributes). MapsLanes may have left and / or right lane changes and / or other information that can be stored in the MapLaneChange structure. The MapLaneChange structure can be used to calculate the SegmentChange structure.

[0116] Connection information can be encoded in a SegmentConnection structure.

[0117] Lane change information can be encoded in a SegmentChange structure.

[0118] Cross-segmentation can be encoded in a segmentation-contention structure.

[0119] Wait condition information can be stored in a SegmentWaitCondition structure.

[0120] Figure 9B This illustrates that some embodiments according to this disclosure may consist of a lane collector 902, a segmentation map generator 904, and / or Figure 9AA flowchart of method 910 implemented by other components of architecture 900. Each block of method 910 and / or other methods described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services, or managed services (standalone or in combination with another managed service) or plug-ins to another product, to name a few. In some implementations, method 910 and / or other methods described herein are stored on and / or executed by a vehicle computing system, such as vehicle computing system 120 described elsewhere herein.

[0121] Generally, method 910 may involve segment graph generator 904 using MapsLane and / or LocalLayoutLane collected by lane collector 902 and stitching them together into a list of unique LocalMapSegments to generate a coarse map or SegmentGraph.

[0122] More specifically, method 910 includes selecting a seed lane in box 912. In the example implementation, the seed lane is selected by analyzing the lanes within the equipment horizontal line (e.g., MapsLane, LocalLayoutLane) until a lane that does not include any entering lanes is identified, and then the identified lane is selected as the seed lane. More generally, the seed lane can be selected in box 912 based on any suitable criteria.

[0123] In box 914, method 910 includes collecting connected lanes in the reverse direction. In an example implementation, box 914 includes collecting lanes within the horizontal line of the equipment that are progressively connected to the seed lane in the reverse direction until a stopping condition is met.

[0124] In block 916, method 910 includes collecting connected lanes in the forward direction. In an example implementation, block 916 includes collecting lanes within a horizontal line of equipment that are progressively connected to a seed lane in the forward direction until a stopping condition is met. The stopping condition at block 916 may be the same as or different from the stopping condition at block 914. Optionally or additionally, the stopping condition for collecting lanes in the rearward and / or forward directions may include one or more of the following:

[0125] - MapsLane and / or LocalLayoutLane participate in the junction path. The junction path may include the junction of one MapsLane or LocalLayoutLane with another MapsLane or LocalLayoutLane (e.g., self-lane and connected merging lanes). Each MapsLane and / or LocalLayoutLane participating in the junction path belongs to the corresponding LaneSegment generated for the associated junction path.

[0126] -MapsLane and / or LocalLayoutLane connections have more than one connected lane.

[0127] - Connecting MapsLane and / or LocalLayoutLane allows for bidirectional travel.

[0128] - The map connecting lanes is not in the current LocalLayout (e.g., outside the equipment's horizontal line).

[0129] In box 918, method 910 includes concatenating the selected and collected MapsLane and / or LocalLayoutLane (e.g., the seed lane selected in box 912 and the lanes collected in boxes 914 and 916) into a LaneSegment. In the example implementation, MapsLane and / or LocalLayoutLane are concatenated according to the following precedent:

[0130] - The MapsLane and / or LocalLayoutLane involved in the intersection path are concatenated into the LaneSegment, and metadata such as competitor segments, waiting elements, and traffic rules are recorded.

[0131] Paths in a LocalLayout that have MapsLane and / or LocalLayoutLane and whose inbound / outbound MapsLane and / or LocalLayoutLane are zero are concatenated into their respective LaneSegments.

[0132] Any unprocessed MapsLane and / or LocalLayoutLane left in the lane buffer are concatenated into their respective LaneSegment.

[0133] In box 918, after concatenating MapsLane and / or LocalLayoutLane into LaneSegment, some implementations can collect metadata such as segmentation relationship information and segmentation change information.

[0134] In at least one example, the segment graph generator 904 may implement or facilitate the implementation of one or more of the following functions: The `updateSegmentGraph` function can be constructed using a valid `LocalLayout`, which can then be fed to the lane collector 902 to collect `LocalLayoutLane`s that can be used to construct the `SegmentGraph`. In some embodiments, the `updateSegmentGraph` function is implemented in the public API of the segment graph generator 904. The `computeSegmentGraph` function can consume the `LaneBuffer` filled by the lane collector 902 and construct the `SegmentGraph`. The `computeSegmentGraph` function may be a private function of the segment graph generator 904 and / or may be called from the `updateSegmentGraph` function. The `computeSegment` function can compute a new unique segment in the current `LocalLayout`; the input `LocalLayoutLane` can be used as a seed lane and can extend in both backward and forward directions until a stopping condition is reached, as described elsewhere in this document. The `computeSegment` function may be a private function of the segment graph generator 904 and / or may be called from the `computeSegmentGraph` function. The `collectSegmentLanes` function collects the MapLane of a segment in a given direction (defined by the StitchDirection), with associated data stored in a MapsLane structure; the segment's MapsLane is collected from a pool of MapsLane in the LaneBuffer (e.g., the output of lane collector 902). For a given segment, the `collectSegmentLanes` function can be called twice from the `computeSegment` function to stretch the segment in both the backward and forward directions. After the SegmentGraph is computed, the `updateSegmentGraphInfo` function can be called from the `updateSegmentGraph` function. The `updateSegmentGraphInfo` function helps collect metadata by iterating through all segments in the SegmentGraph. The `getSegmentConnections` function collects the segment connections of the SegmentGraph. The `getSegmentConnections` function can be private and / or can be called from the `updateSegmentGraphInfo` function.The `getSegmentChanges` function collects all segment changes to the SegmentGraph. The `getSegmentChanges` function can be private and / or can be called from the `updateSegmentGraphInfo` function.

[0135] Lane graph generator 906 can construct an exhaustive lane graph by performing a depth-first search on the LocalMapSegment graph. In some implementations, the updateLaneGraph function can be implemented in the public API of lane graph generator 906. The updateLaneGraph function updates the SegmentGraph by calling the updateSegmentGraph function and constructing it with a new LocalLayout by computing the decomposed LaneGraph lanes through a depth-first search on the new SegmentGraph. Thus, lane graph generator 906 can generate a lane graph that includes the decomposed lane graph lanes and associated metadata, each decomposed lane graph lane having a unique, stable ID to track lane graph lanes between frames. The decomposed LaneGraph can be used to produce an output structure similar to a WM frame, for example, as used by WM generator 908. Self-lane cues can be determined based on the position of the mount in the LaneGraph geometry. The LaneGraph lane IDs can be used with location queries to track stable self-lanes.

[0136] The IDs of the decomposed lanes can be calculated based on the participating MapsLanes. An ID manager can be used to calculate unique IDs for LaneGraph lanes. The ID manager can maintain a hash table (or hash graph) where a list of MapsLane IDs serves as the keys and the LaneGraph lane IDs as the corresponding values. Therefore, the IDs of the MapsLanes participating in lane decomposition can be used as keys to maintain consistent LaneGraph lane IDs across frames.

[0137] In some embodiments, WM keyframes impose triplet point constraints on lane chunks, such that for a point at the center of a lane chunk, there exists a corresponding left-right separator point. However, this is not a constraint on high-resolution maps. For stitched lane channels in WM keyframes, triplets can be generated by projecting points from the other two lines onto each polyline and merging them in a 3-way fashion; this can occur in a loop across all three polylines in the lane channel.

[0138] Consistent with the above, Figure 9CA method 920 for generating WM frames according to some embodiments of this disclosure is illustrated. At 922, the vehicle computing system 120 requests a lane map from a lane map generator 906. At 924, the lane map generator updates the LocalLayout. At 926, the lane map generator requests a SegmentGraph from a SegmentGraph generator 904. At 928, the SegmentGraph generator 904 requests lanes from a lane collector 902 in the LocalLayout. At 930, the lane collector 902 collects lanes from the LocalLayout and populates them into a LaneBuffer 932. At 934, the SegmentGraph generator 904 obtains connection information from the LocalLayout. At 936, the SegmentGraph generator 904 concatenates intersection paths. At 938, the SegmentGraph generator 904 concatenates the collected lanes into mutually exclusive lane segments (LaneSegments). At 940, the SegmentGraph generator 904 obtains the metadata of the LaneSegment. At 942, the segmented map generator 904 returns the segmented map (and associated metadata) to the lane map generator 906. At 944, the lane map generator calculates the decomposed lane map. At 946, the lane map generator 906 obtains the metadata of the lanes in the lane map. At 948, the lane map generator 906 returns the lane map (and associated metadata) to the vehicle computing system 120. At 950, the vehicle computing system requests a WM frame from the WM generator 908. At 952, the WM generator 908 generates a WM frame. At 954, the WM generator 908 returns the WM frame to the vehicle computing system 120.

[0139] Figure 10 The illustration depicts an example lane map calculation from a simple high-definition map 1002 according to some embodiments of the present disclosure. The HD map 1002 includes road segments with MapsLane A to R. A lane collector 902 collects MapsLanes in a LocalLayout. In this example, it is assumed that each of MapsLane A to R is within a LocalLayout and is collected by the lane collector 902.

[0140] The segmented graph generator 904 can then concatenate MapsLane A to R into mutually exclusive LaneSegments LS-A to LS-F, producing SegmentGraph 1004. For example, MapsLane A, B, and C can be concatenated into LaneSegment LS-A, MapsLane G, H, and I might be concatenated into LaneSegment LS-B, and so on. SegmentGraph 1004 and / or associated metadata can identify one or more of the previous LaneSegment, the next LaneSegment, and competing LaneSegments for each LaneSegment.

[0141] Finally, lane graph generator 906 can construct all lane combinations from lane segment LS-A to LS-F to generate lane graph 1006, which includes all path combinations of lane segments as lane graph lanes LGL-A to LGL-E. The first LaneGraph lane LGL-A, including LaneSegments LS-C, LS-A, and LS-D, is... Figure 10 The middle line represents the second LaneGraph lane LGL-B, which includes LaneSegment LS-B, LS-A, and LS-D. Figure 10 The middle line represents a dashed line. The third LaneGraph lane LGL-C, including LaneSegment LS-C, LS-A, and LS-E, is located in... Figure 10 The middle line is represented by a solid line. The fourth LaneGraph lane, LGL-D, including LaneSegments LS-B, LS-A, and LS-E, is located in... Figure 10 The middle line is represented by a long dashed line. This includes the fifth LaneGraph lane, LGL-E, which is part of LaneSegment LS-F. Figure 10 The middle character is represented by a double-dotted line. Figure 10 In other LaneGraphs with overlapping sections, the overlapping sections of LaneGraph lanes are spatially offset from each other for visual perceptibility. In practice, the overlapping sections of LaneGraph lanes within a LaneGraph may not be spatially separated.

[0142] Figure 11A and Figure 11B The following diagram illustrates a mobile autonomous vehicle or equipment 1102 according to some embodiments of the present disclosure. Figure 10 Example lane map calculation for high-definition map 1002. When equipment 1102 is in Figure 11AWhen the equipment 1102 is at one of the three different locations shown on HD map 1002, the lane map calculations are performed at three different times or frames, labeled Frame 0, Frame 1, and Frame 2. In Frame 0, the equipment 1102 is located on MapsLane H, traveling in the direction from MapsLane I to MapsLane G. In Frame 1, the equipment 1102 is located on MapsLane B, traveling in the direction from MapsLane C to MapsLane A. In Frame 2, the equipment 1102 is located on MapsLane K, traveling in the direction from MapsLane I to MapsLane G.

[0143] As shown in the figure, equipment 1102 has a LocalLayout 1104 that updates as equipment 1102 moves. In frame 0, LocalLayout 1104 includes MapsLane G, H, I, C, and D. In frame 1, LocalLayout 1104 includes MapsLane A, B, C, P, Q, and R. In frame 2, LocalLayout 1104 includes MapsLane A, O, J, K, and L.

[0144] In each frame, lane collector 902 collects all MapsLane in LocalLayout 1104. In some implementations, such as Figure 11A As shown, the lane collector 902 collects all MapsLane in the LocalLayout by incrementally growing the set 1106 of valid lanes surrounding the device 1102 within the LocalLayout 1104. More specifically, at each frame, the lane collector 902 can retain each MapsLane from the previous frame that remained within the horizontal line of the LocalLayout 1104 or the device 1102 in the set 1106, can add each new MapsLane within the horizontal line of the LocalLayout 1104 or the device 1102 to the set 1106, and remove from the set 1106 any MapsLane retained or added from the previous frame that was no longer within the horizontal line of the LocalLayout 1104 or the device 1102. For example, at frame 0, which has no previous frame, each of MapsLane G, H, I, C, and D is added to the set 1106 of valid lanes. As another example, in frame 1, MapsLane C remains in set 1106, MapsLane A, B, C, P, Q, and R are newly appearing within the horizontal line of LocalLayout 1104 or Equipment 1102, and are therefore added to set 1106, while MapsLane G, H, I, and D, which were included in the previous frame (frame 0), are removed from set 1106 of the valid lanes. Figure 11AThe image shows invalid lane 1108. As another example, at frame 2, MapsLane A remains in set 1106, MapsLane O, L, K, and J are recently within the horizontal line of LocalLayout 1104 or Equipment 1102, and are therefore added to set 1106. MapsLane B, C, P, Q, and R, which were included in the previous frame (frame 1), are removed from the set of valid lanes 1106. Figure 11A The lane number 1108 is displayed as invalid.

[0145] In some implementations, the lane buffer can be implemented as a dual buffer, including a first effective lane buffer and a second effective lane buffer for collecting valid lanes. For example, the first effective lane buffer may be filled with the currently incrementally growing set of valid lanes during a frame, while the second effective lane buffer stores the previously incrementally growing set of valid lanes. In the next frame, the roles of the first and second effective lane buffers can be switched, such that the second effective lane buffer may be filled with the new currently incrementally growing set of valid lanes during the next frame, while the first effective lane buffer stores the previously incrementally growing set of valid lanes (e.g., the previous currently incrementally growing set of valid lanes). The incremental growth of the set of valid lanes and the use of dual effective lane buffers can allow the generation of a segment graph and / or a lane graph from the previously incrementally growing set of valid lanes, while simultaneously incrementally growing the current set of valid lanes.

[0146] Figure 11BThis describes the SegmentGraph generated at each frame. For example, at frame 0, the segment graph generator 904 can concatenate MapsLane G, H, and I to LaneSegment 1106, MapsLane C to LaneSegment 1108, and MapsLane D to LaneSegment 1110 to generate the SegmentGraph shown at frame 0. At frame 1, the segment graph generator 904 can concatenate MapsLane A, B, and C to LaneSegment 1112 and MapsLane P, Q, and R to LaneSegment 1114 to generate the SegmentGraph shown at frame 1. At frame 2, the segment graph generator 904 can concatenate MapsLane A to LaneSegment 1116, MapsLane O to LaneSegment 1118, and MapsLane J, K, and L to LaneSegment 1120 to generate the SegmentGraph shown at frame 2. Although Figure 11B Not shown in the figure, but the SegmentGraph and / or associated metadata at each frame can identify one or more of the previous LaneSegment, the next LaneSegment, and the competing LaneSegment for each LaneSegment.

[0147] Figure 11B The diagram also illustrates the LaneGraph generated at each frame. For example, at frame 0, lane graph generator 906 can organize all lane combinations from lane segments 1106, 1108, and 1110 to generate the lane graph shown at frame 0, which includes all path combinations from lane segments 1106, 1108, and 1110, such as LaneGraph lanes 1122 and 1124. At frame 1, lane graph generator 906 can organize all lane combinations from lane segments 1112 and 1114 to generate the lane graph shown at frame 1, including all path combinations from lane segments 1112 and 1114 as lane graph lanes 1126 and 1128. At frame 2, lane map generator 906 can organize all lane combinations from lane segments 1116, 1118, and 1120 to generate the lane map shown at frame 2, including all path combinations from lane segments 1116, 1118, and 1120 as LaneGraph lanes 1130 and 1132.

[0148] Figure 11BAlso included is illustration 1134, which shows different types of LaneGraph lanes that can be included in the LaneGraph at one or more frames. The LaneGraph lane on which the vehicle or machine corresponding to equipment 1102 is currently traveling is called an "ego-lane". LaneGraph lanes 1122, 1126, and 1132 are examples of ego-lanes. LaneGraph lanes merged with ego-lanes are called "merged lanes". LaneGraph lane 1124 is an example of a merged lane. LaneGraph lanes split from ego-lanes are called "split lanes". LaneGraph lane 1130 is an example of a split lane. LaneGraph lanes crossing ego-lanes are called "competitor lanes". LaneGraph lane 1128 is an example of a competitor lane.

[0149] For safe navigation of autonomous vehicles, WM frames are typically computed at a rate of approximately 30 WM frames per second. The large data volume of WM frames consumes significant communication, processing, and / or storage resources on the autonomous vehicle. Therefore, some embodiments herein separate the generation of data (e.g., WM frames) for the autonomous vehicle into multiple tasks, where tasks with higher latency are performed less frequently than those with lower latency. For example, data generation can be divided into higher-latency, lower-frequency WM keyframe generation tasks and lower-latency, higher-frequency transformation tasks. Alternatively or additionally, tasks may include lower-latency, higher-frequency transformation computation tasks. These tasks may be repeated or iterated over time. For example, WM keyframes may be computed using a first frequency, while other WM frames (e.g., non-keyframes) may be computed using a second frequency greater than the first frequency.

[0150] In this and other implementations, the WM generator 908 can be divided into two, three, or some other number of nodes, where each node processes different tasks in the task according to the corresponding frequency. Figure 12 An example implementation of a WM generator 908 having three nodes 1202, 1204, and 1206 according to some embodiments of the present disclosure is illustrated. The three nodes 1202, 1204, and 1206 specifically include a WM generator 1202, a WM pose calculation node 1204, and a WM converter 1206.

[0151] Typically, the WM generator 908 can receive lane map 1208 and positioning result 1210 as input, and can output WM frame 1212 to one or more WM consumers 1213. Each positioning result 1210 can identify the current position of the autonomous vehicle relative to the WM frame 1212. The WM consumer 1213 may include, for example, a vehicle computing system 120, which can use or consume the WM frame 1212 to control the operation of the corresponding autonomous vehicle 150 or equipment. Alternatively or additionally, the WM consumer 1213 may include some other systems, devices, applications, etc. that use or consume the WM frame 1212.

[0152] WM generator 1202 can be a high-latency (compared to nodes 1204 and 1206) and low-frequency (compared to nodes 1204 and 1206) node, while WM pose calculation node 1204 and WM converter 1206 can be low-latency (compared to node 1202) and high-frequency (compared to node 1202) nodes.

[0153] Typically, the WM generator 1202 can compute the WM keyframe 1214 from the LaneGraph 1208. The computation of the WM keyframe can be a relatively high-latency computation, which can be performed by the WM generator 1202 at a first frequency (e.g., approximately 10 Hz). According to some implementations, the WM generator 1202 implements a dual buffer for computing the WM keyframe 1214, wherein it alternately fills one of two buffers 1216, 1218 with the current WM keyframe while the preceding (e.g., immediately preceding) WM keyframe 1214 is stored in the other of the two buffers 1216, 1218. Buffers 1216, 1218 may optionally be referred to as world model buffers to distinguish them from other buffers described herein. The WM pose computation node 1204 and the WM transformer node 1206 can read the buffers 1216, 1218 that store the previous WM keyframe 1214 at any given time while performing their respective tasks. When the calculation of the current WM keyframe is complete, the current WM keyframe becomes the previous WM keyframe 1214 in the corresponding one of buffers 1216 and 1218, and a new current WM keyframe can be calculated by filling the other of the two buffers 1216 and 1218 with the new current WM keyframe. In this way, each of buffers 1216 and 1218 can alternately store the previous WM keyframe 1214 or fill it with the current WM keyframe during the calculation of the current WM keyframe.

[0154] WM pose calculation node 1204 can utilize a second frequency and calculate the effective transformation 1220 of the preceding WM keyframe 1214 for each WM frame from the positioning result 1210 and the preceding WM keyframe 1214. The positioning result 1210 can include the current position of the equipment at any given time and for any given WM frame generated by WM generator 908. WM pose calculation node 1204 can use the current position of the equipment to determine the corresponding effective transformation 1220 (e.g., a local map to equipment transformation matrix) that transforms the preceding WM keyframe 1214 into the equipment's coordinate system. The second frequency can be greater than the first frequency, allowing multiple effective transformations 1220 to be calculated sequentially for a given preceding WM keyframe 1214. In an example where the first frequency is 10 Hz and the second frequency is 30 Hz, WM pose calculation node 1204 can calculate three consecutive effective transformations 1220 for each preceding WM keyframe 1214.

[0155] WM Transformer 1206 can use valid transform 1220 to transform the preceding WM keyframe 1214 to the equipment's coordinate system. For example, WM Transformer 1206 can use sequential valid transform 1220 to sequentially transform the preceding WM keyframe 1214 to the equipment's coordinate system to generate sequential WM frames 1212. WM Transformer 1206 can perform transforms at a second frequency, such that any given preceding WM keyframe 1214 can be sequentially transformed multiple times to generate multiple sequential WM frames 1212.

[0156] Figure 13 This is a flowchart illustrating a method 1300 for generating WM frames according to some embodiments of the present disclosure. Each block of method 1300 and / or other methods described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services, or managed services (standalone or in combination with another managed service) or plug-ins to another product, to name a few. In some embodiments, method 1300 and / or other methods described herein are stored on and / or executed by a vehicle computing system, such as vehicle computing system 120 described elsewhere herein. For example, method 1300 can be executed at least in part by an autonomous vehicle 150 or a WM generator 908 equipped on vehicle computing system 120.

[0157] In box 1302, method 1300 includes calculating the current WM keyframes around the autonomous vehicle based on map data. For example, box 1302 may include WM generator 1202 of WM generator 908 calculating the current WM keyframes from the current LaneGraph 1208.

[0158] In block 1304, method 1300 includes transforming a previous WM keyframe to the autonomous vehicle's coordinate system at a first moment before the calculation of the current WM keyframe is completed, to generate a first WM frame. For example, block 1304 may include WM transformer 1206 of WM generator 908 transforming a previous WM keyframe 1214 to the autonomous vehicle's coordinate system at a first moment before and during the calculation of the current WM keyframe, to generate a first WM frame 1212.

[0159] In block 1306, method 1300 includes transforming a previous WM keyframe to the autonomous vehicle's coordinate system at a second time, after the first time and before the calculation of the current WM keyframe is completed, to generate a second WM frame. For example, block 1306 may include WM transformer 1206 of WM generator 908 transforming the same previous WM keyframe 1214 to the autonomous vehicle's coordinate system at a second time, after the first time and during and before the completion of the calculation of the current WM keyframe, to generate a second WM frame 1212.

[0160] In some implementations, method 1300 may further include determining a transformation to be applied to the previous WM keyframe. In this example, transforming the previous WM keyframe to the autonomous vehicle's coordinate system may include transforming the previous WM keyframe to the autonomous vehicle's coordinate system according to the determined transformation. For example, the WM pose calculation node 1204 of the WM generator 908 may determine, for example, an effective transformation 1220 calculated in the previous WM keyframe 1214 based on the positioning result 1214, and may transform the previous WM keyframe 1214 to the autonomous vehicle's coordinate system according to the effective transformation 1220.

[0161] The map data used to calculate the current WM keyframe can include, for example, a lane map, such as any lane map described herein. Therefore, calculating the current WM keyframe around the autonomous vehicle based on map data can include calculating the current WM keyframe around the autonomous vehicle based on the lane map. Some embodiments of method 1300 may further include generating a lane map from a portion of an HD map, which includes incrementally growing the set of valid lanes around the autonomous vehicle from the portion of the HD map, such as regarding, for example... Figure 11A As described. For example, the incrementally growing set of valid lanes can be stored in a first valid lane buffer, and method 1300 may also include filling a second valid lane buffer with the currently incrementally growing set of valid lanes.

[0162] Alternatively or additionally, previous WM keyframes may be stored in a first WM buffer, and method 1300 may further include filling a second WM buffer with the current WM keyframe when calculating the current WM keyframe. As an example, the first WM buffer may include... Figure 12 The buffer 1216, and the second WM buffer may include Figure 12 The buffer 1218. In some embodiments, after calculating the current WM keyframe, the current WM keyframe can be stored in a second WM buffer as a new previous WM keyframe, and the method 1300 may further include, after calculating the current WM keyframe, calculating subsequent WM keyframes around the autonomous vehicle based on additional map data (e.g., an updated lane map), filling the first WM buffer with the subsequent WM keyframes when calculating the subsequent WM keyframes, transforming the new previous WM keyframe stored in the second WM buffer to the coordinate system of the autonomous vehicle at a third time before the calculation of the subsequent WM keyframes is completed, and transforming the new previous WM keyframe stored in the second WM buffer to the coordinate system of the autonomous vehicle at a fourth time after the third time and before the calculation of the subsequent WM keyframes is completed.

[0163] Figure 14 This is a flowchart illustrating another method 1400 for generating WM frames according to some embodiments of this disclosure. Each block of method 1400 and / or other methods described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services, or managed services (standalone or in combination with another managed service) or plug-ins to another product, to name a few. In some implementations, method 1400 and / or other methods described herein are stored on and / or performed by a vehicle computing system, such as vehicle computing system 120 described elsewhere herein. For example, method 1400 can be performed at least in part by an autonomous vehicle 150 or a WM generator 908 equipped on vehicle computing system 120.

[0164] In box 1402, method 1400 includes calculating WM keyframes around the autonomous vehicle over time based on map data at a first frequency. For example, box 1302 may include WM generator 1202 of WM generator 908 sequentially calculating current WM keyframes from a sequence of current LaneGraph 1208 at a first frequency.

[0165] In block 1404, method 1400 includes transforming the most recent WM keyframe to the autonomous vehicle's coordinate system over time at a second frequency greater than a first frequency to generate WM frames, such that multiple WM frames are generated from each WM keyframe. For example, block 1404 may include a WM transformer 1206 of a WM generator 908 transforming sequential previous (or most recent) WM keyframes 1214 to the autonomous vehicle's coordinate system over time at a second frequency to generate WM keyframes 1212, such that multiple WM keyframes 1212 are generated from each previous WM keyframe 1214.

[0166] In some implementations, method 1400 may further include determining, over time and at a second frequency, a transformation applied to the most recent WM keyframe. In this example, transforming the most recent (or previous) WM keyframe to the autonomous vehicle's coordinate system over time may include transforming the most recent WM keyframe to the autonomous vehicle's coordinate system according to the determined transformation. For example, the WM pose calculation node 1204 of the WM generator 908 may determine, over time, based on the positioning result 1214, for example, calculating an effective transformation 1220 for the previous WM keyframe 1214, and transforming the previous WM keyframe 1214 to the autonomous vehicle's coordinate system according to the effective transformation 1220.

[0167] The map data on which the WM keyframes are calculated can include, for example, a lane map, such as any lane map described herein. Therefore, calculating WM keyframes around an autonomous vehicle based on map data can include calculating WM keyframes around the autonomous vehicle based on a lane map. Some embodiments of method 1400 may further include generating a lane map from a portion of an HD map, including incrementally growing the set of valid lanes around the autonomous vehicle from the portion of the HD map, such as regarding, for example... Figure 11A As described. Some embodiments of method 1400 may further include alternately filling the first and second effective lane buffers with a currently increasing set of effective lanes. In this and other embodiments, the first effective lane buffer may be filled with the currently increasing set of effective lanes when the most recently increasing set of effective lanes is stored in the second effective lane buffer, and the second effective lane buffer may be filled with the currently increasing set of effective lanes when the most recently increasing set of effective lanes is stored in the first effective lane buffer.

[0168] Alternatively or additionally, method 1400 may also include alternately filling a first WM buffer and a second world model buffer with WM keyframes. As an example, the first WM buffer may include... Figure 12 The buffer 1216, and the second WM buffer may include Figure 12Buffer 1218. When the most recent WM keyframe is stored in the second WM buffer, the first WM buffer can be filled with the currently computed WM keyframe. When the most recent WM keyframe is stored in the first WM buffer, the second WM buffer can be filled with the currently computed WM keyframe.

[0169] The compression performance according to some embodiments described herein may be significant. It can be assumed that the size of each LaneGraph is significantly larger than the size of each transform. In this example, the compression ratio R... c It can simply be the transmission interval of a LaneGraph or lane graph interval frame with static geometry. The transmission interval laneGraphIntervalFrame may be related to the size of the road segment (roadSegmentLength) and the vehicle speed (vehicleSpeed). Specifically, the transmission interval laneGraphIntervalFrame can be determined according to Equation 1:

[0170]

[0171] Assuming a road segment length of 20 meters and a vehicle speed of 90 miles per hour (or approximately 40 meters per second), the compression ratio R... c It might be approximately (20 / 40)*20 = 10. Assuming a vehicle speed of 22 mph (or approximately 10 meters per second) and other conditions similar to the previous example, the compression ratio R... c It could be approximately (20 / 10)*20 = 40. Assuming traffic stops and resumes, and conditions similar to the previous example, the compression ratio R... c It can be 100 or higher.

[0172] Therefore, in any one or both of methods 1300, 1400 or other methods herein, the compression ratio of the WM keyframes calculated at the first and second frequencies may depend on the speed of the autonomous vehicle and may be at least 10, at least 40, at least 100 or some other value.

[0173] Example autonomous vehicles

[0174] Figure 15AThis is an illustration of an example autonomous vehicle 1500 according to some embodiments of the present disclosure. The autonomous vehicle 1500 (which may alternatively be referred to herein as "vehicle 1500") may include, but is not limited to, passenger vehicles such as automobiles, trucks, buses, ambulances, shuttles, electric or motorized bicycles, motorcycles, fire trucks, police cars, ambulances, boats, engineering vehicles, underwater vehicles, drones, and / or other types of vehicles (e.g., driverless and / or capable of accommodating one or more passengers). Autonomous vehicles are generally described according to the level of automation defined by the National Highway Traffic Safety Administration (NHTSA) of the U.S. Department of Transportation and the Society of Automotive Engineers (SAE) "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 1500 may be able to have one or more of the capabilities according to Level 3-Level 5 of the autonomous driving level. For example, depending on the embodiment, vehicle 1500 may have conditional automation (level 3), high automation (level 4), and / or full automation (level 5).

[0175] Vehicle 1500 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 1500 may include a propulsion system 1550, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 1550 may be connected to the drivetrain of vehicle 1500, which may include a transmission, to enable propulsion of vehicle 1500. Propulsion system 1550 may be controlled in response to receiving a signal from throttle / accelerator 1552.

[0176] A steering system 1554, which may include a steering wheel, can be used to steer the vehicle 1500 (e.g., along a desired path or route) when the propulsion system 1550 is operating (e.g., when the vehicle is in motion). The steering system 1554 may receive signals from the steering actuator 1556. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0177] The brake sensor system 1546 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1548 and / or the brake sensor.

[0178] It may include one or more CPUs, System-on-a-Chip (SoC) 1504 ( Figure 15COne or more controllers 1536, including one or more GPUs, may provide signals (e.g., signals representing commands) to one or more components and / or systems of vehicle 1500. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 1548, to operate steering system 1554 via one or more steering actuators 1556, and to operate propulsion system 1550 via one or more throttles / accelerators 1552. One or more controllers 1536 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 1500. One or more controllers 1536 may include a first controller 1536 for autonomous driving functions, a second controller 1536 for functional safety functions, a third controller 1536 for artificial intelligence functions (e.g., computer vision), a fourth controller 1536 for infotainment functions, a fifth controller 1536 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1536 can handle two or more of the functions described above, and two or more controllers 1536 can handle a single function, and / or any combination thereof.

[0179] One or more controllers 1536 may provide signals for controlling one or more components and / or systems of vehicle 1500 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, global navigation satellite system sensors 1558 (e.g., GPS sensors), RADAR sensors 1560, ultrasonic sensors 1562, LIDAR sensors 1564, inertial measurement unit (IMU) sensors 1566 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1596, stereo cameras 1568, wide-angle cameras 1570 (e.g., fisheye cameras), infrared cameras 1572, surround cameras 1574 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1598, speed sensors 1544 (e.g., for measuring the rate of vehicle 1500), vibration sensors 1542, steering sensors 1540, braking sensors (e.g., as part of braking sensor system 1546), and / or other sensor types.

[0180] One or more of the controllers 1536 may receive inputs (e.g., represented by input data) from the instrument cluster 1532 of the vehicle 1500 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1534, an auditory signaling device, a speaker, and / or via other components of the vehicle 1500. These outputs may include information such as vehicle speed, rate, time, map data (e.g., [missing information]). Figure 15C Information such as the HD map 1522, location data (e.g., the location of vehicle 1500 on the map), direction, and the location of other vehicles (e.g., occupying a grid), as well as information about objects and their states perceived by the controller 1536, etc. For example, the HMI display 1534 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0181] The vehicle 1500 further includes a network interface 1524, which can communicate via one or more networks using one or more wireless antennas 1526 and / or a modem. For example, the network interface 1524 may be able to communicate via LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The one or more wireless antennas 1526 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (LPWAN) such as LoRaWAN, SigFox, etc.

[0182] Figure 15B For use in accordance with some embodiments of this disclosure Figure 15A This is an example of the camera position and field of view of an autonomous vehicle 1500. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 1500.

[0183] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 1500. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 920 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-transparent (RCCC) color filter array, a red-transparent-blue (RCCB) color filter array, a red-blue-green (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a high-resolution camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity.

[0184] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0185] One or more of the cameras can be mounted in mounting components such as custom-designed (3-D printed) components to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3-D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0186] A camera with a field of view that includes the environment in front of the vehicle 1500 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 1536 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (“LDW”), Autonomous Cruise Control (“ACC”), and / or other functions such as traffic sign recognition.

[0187] A variety of cameras can be used in front-facing configurations, including monocular camera platforms such as CMOS (Complementary Metal-Oxide-Semiconductor) color imagers. Another example could be a wide-angle camera 1570, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although Figure 15B The middle image shows only one wide-angle camera, but any number of wide-angle cameras 1570 can be present on the vehicle 1500. Furthermore, a remote camera 1598 (e.g., a pair of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The remote camera 1598 can also be used for object detection and classification, as well as basic object tracking.

[0188] One or more stereo cameras 1568 may also be included in a front-mounted configuration. The stereo camera 1568 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (e.g., an FPGA) with an integrated CAN or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 1568 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1568 may be used in addition to those described herein, or alternatively.

[0189] Cameras with a field of view including the side portion of the vehicle 1500 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 1574 (e.g., ... Figure 15B The four surround cameras 1574 shown can be positioned around the vehicle 1500. The surround cameras 1574 can include a wide-angle camera 1570, a fisheye camera, a 360-degree camera, and / or the like. For example, four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1574 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround-view camera.

[0190] A camera with a field of view that includes the environment behind the vehicle 1500 (e.g., a rear-view camera) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 1598, stereo camera 1568, infrared camera 1572, etc.).

[0191] Figure 15C For use in accordance with some embodiments of this disclosure Figure 15A The example autonomous vehicle 1500 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented in hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0192] Figure 15C Each component, feature, and system in vehicle 1500 is illustrated as being connected via bus 1502. Bus 1502 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 1500 used to assist in the control of various features and functions of vehicle 1500, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. CAN bus may be ASIL B compliant.

[0193] Although bus 1502 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 1502 is represented by a single line, this is not intended to be limiting. For example, any number of buses 1502 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1502 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1502 may be used for a collision avoidance function, and a second bus 1502 may be used for drive control. In any example, each bus 1502 may communicate with any component of vehicle 1500, and two or more buses 1502 may communicate with the same component. In some examples, each SoC 1504, each controller 1536, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 1500) and may be connected to a common bus such as the CAN bus.

[0194] Vehicle 1500 may include one or more controllers 1536, such as those described herein. Figure 15A The controllers described herein. Controller 1536 can be used for a wide variety of functions. Controller 1536 can be coupled to any other different components and systems of vehicle 1500 and can be used for the control of vehicle 1500, artificial intelligence of vehicle 1500, infotainment and / or the like for vehicle 1500.

[0195] Vehicle 1500 may include one or more System-on-Chip (SoC) 1504. SoC 1504 may include CPU 1506, GPU 1508, processor 1510, cache 1512, accelerator 1514, data storage 1516, and / or other components and features not shown. SoC 1504 can be used to control vehicle 1500 across a wide variety of platforms and systems. For example, one or more SoCs 1504 may be combined with an HD map 1522 in a system (e.g., the system of vehicle 1500), the HD map being accessible from one or more servers (e.g., via a network interface 1524). Figure 15D One or more servers (1578) receive map refresh and / or updates.

[0196] CPU 1506 may include CPU clusters or CPU complexes (or, alternatively, referred to herein as "CCPLEX"). CPU 1506 may include multiple cores and / or L2 cache. For example, in some embodiments, CPU 1506 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 1506 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., 2MB L2 cache). CPU 1506 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of CPU 1506 can be active at any given time.

[0197] The CPU 1506 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. The CPU 1506 can further implement enhanced algorithms for managing power states, where allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0198] GPU 1508 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). GPU 1508 may be programmable and efficient for parallel workloads. In some examples, GPU 1508 may use an enhanced tensor instruction set. GPU 1508 may include one or more streaming microprocessors, wherein each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, GPU 1508 may include at least eight streaming microprocessors. GPU 1508 may use a computer-based application programming interface (API). Furthermore, GPU 1508 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0199] In automotive and embedded applications, the GPU 1508 can be power-optimized for optimal performance. For example, the GPU 1508 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 1508 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, dispatch units, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations for efficient workload execution. Streaming microprocessors may include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors may include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0200] The GPU 1508 may include, in some examples, a high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem providing peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, synchronous graphics random access memory (SGRAM), such as fifth-generation graphics double data rate synchronous random access memory (GDDR5), may be used.

[0201] The GPU 1508 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 1508 to directly access the CPU 1506 page tables. In such examples, when the GPU 1508 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 1506. In response, the CPU 1506 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 1508. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 1506 and the GPU 1508, simplifying GPU 1508 programming and porting applications to the GPU 1508.

[0202] In addition, the GPU 1508 may include access counters that track how frequently the GPU 1508 accesses the memory of other processors. These access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0203] SoC 1504 may include any number of caches 1512, including those described herein. For example, cache 1512 may include an L3 cache available to both CPU 1506 and GPU 1508 (e.g., it is connected to both CPU 1506 and GPU 1508). Cache 1512 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, but a smaller cache size may also be used.

[0204] SoC 1504 may include an arithmetic logic unit (ALU), which can be used to perform processing of any of a variety of tasks or operations related to vehicle 1500—such as processing a DNN. Additionally, SoC 1504 may include a floating-point unit (FPU)—or other mathematical coprocessor or digital coprocessor type—for performing mathematical operations within the system. For example, SoC 104 may include one or more FPUs integrated as execution units within CPU 1506 and / or GPU 1508.

[0205] SoC 1504 may include one or more accelerators 1514 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 1504 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to supplement GPU 1508 and offload some tasks from GPU 1508 (e.g., freeing up more cycles of GPU 1508 to perform other tasks). As an example, accelerator 1514 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0206] Accelerator 1514 (e.g., a hardware acceleration cluster) may include a Deep Learning Accelerator (DLA). The DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than a general-purpose GPU and significantly outperform CPUs. The TPU can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0207] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0208] The DLA can perform any function of the GPU 1508, and by using inference accelerators, for example, designers can target either the DLA or the GPU 1508 for any function. For instance, a designer can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 1508 and / or other accelerators 1514.

[0209] Accelerator 1514 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0210] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0211] DMA enables PVA components to access system memory independently of the CPU 1506. DMA can support any number of features to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0212] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., a VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0213] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. Consequently, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelization. For example, in some embodiments, two or more vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0214] Accelerator 1514 (e.g., a hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 1514. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks accessible by both PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. PVA and DLA may access memory via a backbone that provides high-speed memory access to PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network interconnecting PVA and DLA to memory.

[0215] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0216] In some examples, the SoC 1504 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0217] Accelerators 1514 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are efficient in object detection and integer mathematical operations.

[0218] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0219] In some examples, PVA can be used to perform intensive optical flow. For instance, PVA can be used to process raw RADAR data (e.g., using 4D Fast Fourier Transform) before emitting the next RADAR pulse to provide a processed RADAR signal. In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0220] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run neural networks to regress the confidence value. The neural network can take at least some subset of parameters as its input, such as bounding box dimensions, ground plane estimates obtained (e.g. from another subsystem), outputs from inertial measurement unit (IMU) sensors 1566 related to the orientation and distance of vehicle 1500, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 1564 or RADAR sensor 1560), etc.

[0221] SoC 1504 may include one or more data storage units 1516 (e.g., memory). The data storage unit 1516 may be on-chip memory of SoC 1504, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, the data storage unit 1516 may be large enough to store multiple instances of the neural network. The data storage unit 1516 may include L2 or L3 cache 1512. References to the data storage unit 1516 may include references to memory associated with the PVA, DLA, and / or other accelerators 1514 as described herein.

[0222] SoC 1504 may include one or more processors 1510 (e.g., embedded processors). Processor 1510 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 1504 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 1504 thermal and temperature sensor management, and / or SoC 1504 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 1504 may use the ring oscillator to detect the temperature of CPU 1506, GPU 1508, and / or accelerator 1514. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 1504 into a lower power state and / or place vehicle 1500 into a driver-safe parking mode (e.g., safely stop vehicle 1500).

[0223] The processor 1510 may further include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces, as well as a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0224] The processor 1510 may further include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0225] The processor 1510 may further include a security cluster engine, which includes a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0226] The processor 1510 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0227] The processor 1510 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0228] Processor 1510 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 1570, the surround camera 1574, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0229] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0230] The video image compositer can also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 1508 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 1508 is powered on and actively performing 3D rendering, the video image compositer can be used to offload the GPU 1508 to improve performance and responsiveness.

[0231] The SoC 1504 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions for receiving video and input from a camera. The SoC 1504 may further include an input / output controller that can be software-controlled and can be used to receive I / O signals not assigned to a specific role.

[0232] SoC 1504 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 1504 can be used to process data from cameras and sensors (e.g., LIDAR sensor 1564, RADAR sensor 1560, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 1502 (e.g., vehicle 1500 speed, steering wheel position, etc.), and data from GNSS sensor 1558 (connected via Ethernet or CAN bus). SoC 1504 may further include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine, and which can be used to free up CPU 1506 from routine data management tasks.

[0233] The SoC 1504 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 1504 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 1506, GPU 1508, and data storage 1516, the accelerator 1514 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0234] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0235] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1520) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could further include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0236] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions," along with a light, can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network, which informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 1508.

[0237] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 1500. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 1504 provides security against theft and / or carjacking.

[0238] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 1596 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 1504 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 1558. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 1562, the control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0239] The vehicle may include a CPU 1518 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1504 via a high-speed interconnect (e.g., PCIe). The CPU 1518 may include, for example, an x86 processor. The CPU 1518 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 1504, and / or monitoring the status and health of the controller 1536 and / or the infotainment SoC 1530.

[0240] Vehicle 1500 may include a GPU 1520 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 1504 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 1520 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based on inputs (e.g., sensor data) from sensors of vehicle 1500.

[0241] Vehicle 1500 may further include a network interface 1524, which may include one or more wireless antennas 1526 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 1524 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 1578 and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 1500 with information about vehicles approaching vehicle 1500 (e.g., vehicles in front, to the side, and / or behind vehicle 1500). This functionality may be part of vehicle 1500's cooperative adaptive cruise control function.

[0242] Network interface 1524 may include a SoC that provides modulation and demodulation functions and enables controller 1536 to communicate via a wireless network. Network interface 1524 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0243] Vehicle 1500 may further include data storage 1528, which may include off-chip (e.g., off-chip SoC 1504) storage devices. Data storage 1528 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0244] Vehicle 1500 may further include GNSS sensors 1558 (e.g., GPS and / or auxiliary GPS sensors) for auxiliary mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1558 may be used, including, for example, but not limited to, GPS sensors using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0245] Vehicle 1500 may further include a RADAR sensor 1560. The RADAR sensor 1560 can be used by vehicle 1500 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level can be ASILB. The RADAR sensor 1560 can use CAN and / or bus 1502 (e.g., to transmit data generated by the RADAR sensor 1560) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 1560 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0246] The RADAR sensor 1560 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, the long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. The RADAR sensor 1560 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assistance and forward collision warning. The long-range RADAR sensor can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 1500's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 1500's lane.

[0247] As an example, a mid-range RADAR system can include a range of up to 960m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 1550 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0248] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0249] Vehicle 1500 may further include ultrasonic sensors 1562. Ultrasonic sensors 1562, which may be positioned at the front, rear, and / or sides of vehicle 1500, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 1562 can be used, and different ultrasonic sensors 1562 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 1562 can operate at functional safety level ASIL B.

[0250] Vehicle 1500 may include a LIDAR sensor 1564. The LIDAR sensor 1564 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 1564 may be of functional safety level ASIL B. In some examples, vehicle 1500 may include multiple LIDAR sensors 1564 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0251] In some examples, the LiDAR sensor 1564 may be able to provide a list of objects and their distances within a 360-degree field of view. Commercially available LiDAR sensors 1564 may have an advertising range of, for example, approximately 100m, with an accuracy of 2cm-3cm, and support for 100Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 1564 may be used. In such examples, the LiDAR sensor 1564 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 1500. In such examples, the LiDAR sensor 1564 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. Front-mounted LiDAR sensors 1564 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0252] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. A flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using a flash LIDAR, and because a flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 1564 is less susceptible to motion blur, vibration, and / or shock.

[0253] The vehicle may further include an IMU sensor 1566. In some examples, the IMU sensor 1566 may be located at the center of the rear axle of the vehicle 1500. The IMU sensor 1566 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 1566 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 1566 may include an accelerometer, a gyroscope, and a magnetometer.

[0254] In some embodiments, the IMU sensor 1566 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 1566 can enable the vehicle 1500 to estimate heading without input from a magnetic sensor by directly observing and correlating velocity changes from GPS to the IMU sensor 1566. In some examples, the IMU sensor 1566 and the GNSS sensor 1558 can be combined into a single integrated unit.

[0255] The vehicle may include a microphone 1596 placed in and / or around the vehicle 1500. Among other things, the microphone 1596 may be used for emergency vehicle detection and identification.

[0256] The vehicle may further include any number of camera types, including stereo camera 1568, wide-angle camera 1570, infrared camera 1572, surround camera 1574, long-range and / or mid-range camera 1598, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 1500. The types of cameras used depend on the embodiment and the requirements of the vehicle 1500, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1500. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described herein with respect to... Figure 15A and Figure 15B It was described in more detail.

[0257] Vehicle 1500 may further include vibration sensor 1542. Vibration sensor 1542 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 1542 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between a power drive shaft and a free-rotating shaft).

[0258] Vehicle 1500 may include ADAS system 1538. In some examples, ADAS system 1538 may include SoC. ADAS system 1538 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC) and / or other features and functions.

[0259] The ACC system can use a RADAR sensor 1560, a LIDAR sensor 1564, and / or a camera. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 1500 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 1500 to change lanes. Lateral ACC is associated with other ADAS applications such as LC and CWS.

[0260] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or through a network connection (e.g., via the Internet) through network interface 1524 and / or wireless antenna 1526. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 1500 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of these I2V and V2V information sources. Given information about vehicles ahead of vehicle 1500, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0261] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0262] An AEB (Autonomous Emergency Braking) system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0263] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings. When the driver indicates intentional lane departure, the LDW system is deactivated by activating a turn signal. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0264] The LKA system is a variation of the LDW system. If vehicle 1500 begins to leave the lane, the LKA system provides steering input or braking to correct vehicle 1500.

[0265] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses turn signals. The BSW system can utilize a rear-facing camera and / or RADAR sensor 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0266] The RCTW system can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle is reversing. Some RCTW systems include AEB to ensure the application of the vehicle's brakes to avoid a collision. The RCTW system may use one or more rear-mounted RADAR sensors 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0267] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as the ADAS system alerts the driver and allows them to determine whether a safe condition truly exists and take appropriate action. However, in the autonomous vehicle 1500, in the event of conflicting results, the vehicle 1500 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 1536 or the second controller 1536). For example, in some embodiments, the ADAS system 1538 may be a backup and / or auxiliary computer for providing perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 1538 may be provided to a supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0268] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0269] The supervisory MCU can be configured to run a neural network trained and configured to determine the conditions under which the auxiliary computer provides a false alarm based on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include components of and / or be included as components of the SoC 1504.

[0270] In other examples, ADAS system 1538 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware used by the host computer does not cause a substantial error.

[0271] In some examples, the output of ADAS system 1538 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 1538 issues a forward collision warning because an object is immediately in front, the perception block can use this information when recognizing the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0272] Vehicle 1500 may further include an infotainment SoC 1530 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 1530 may include a combination of hardware and software that can be used to provide vehicle 1500 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.) and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 1530 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 1534, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 1530 may further be used to provide information (e.g., visual and / or auditory) to the vehicle's users, such as information from the ADAS system 1538, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0273] The infotainment SoC 1530 may include GPU functionality. The infotainment SoC 1530 can communicate with other devices, systems, and / or components of the vehicle 1500 via bus 1502 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1530 may be coupled to a supervisory MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 1536 (e.g., the primary and / or backup computer of the vehicle 1500). In such an example, the infotainment SoC 1530 may place the vehicle 1500 into a driver-safe parking mode as described herein.

[0274] Vehicle 1500 may further include instrument cluster 1532 (e.g., digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). Instrument cluster 1532 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). Instrument cluster 1532 may include a set of instruments such as speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between infotainment SoC 1530 and instrument cluster 1532. In other words, instrument cluster 1532 may be included as part of infotainment SoC 1530, or vice versa.

[0275] Figure 15D For cloud-based servers and according to some embodiments of this disclosure Figure 15A The following is a system diagram illustrating communication between example autonomous vehicles 1500. System 1576 may include server 1578, network 1590, and vehicles including vehicle 1500. Server 1578 may include two or more GPUs 1584(A)-1584(H) (collectively referred to herein as GPU 1584), PCIe switches 1582(A)-1582(H) (collectively referred to herein as PCIe switch 1582), and / or CPUs 1580(A)-1580(B) (collectively referred to herein as CPU 1580). GPU 1584, CPU 1580, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 1586, such as, but not limited to, NVLink interface 1588 developed by NVIDIA. In some examples, GPU 1584 is connected via NVLink and / or NVSwitch SoC, and GPU 1584 and PCIe switch 1582 are connected via PCIe interconnect. Although the diagram illustrates eight GPUs 1584, two CPUs 1580, and four PCIe switches 1582, this is not intended to be limiting. Depending on the embodiment, each of the servers 1578 may include any number of GPUs 1584, CPUs 1580, and / or PCIe switches 1582. For example, each of the servers 1578 may include eight, sixteen, thirty-two, and / or more GPUs 1584.

[0276] Server 1578 can receive image data from vehicles via network 1590, representing images of unexpected or changed road conditions such as recently commenced roadworks. Server 1578 can transmit neural network 1592, updated neural network 1592, and / or map information 1594, including information about traffic and road conditions, to vehicles via network 1590. Updates to map information 1594 may include updates to HD map 1522, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 1592, updated neural network 1592, and / or map information 1594 may have been generated from new training and / or data received from any number of vehicles in the environment and / or based on experience from training performed at a data center (e.g., using server 1578 and / or other servers).

[0277] Server 1578 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more categories of machine learning techniques, including but not limited to: categories such as supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to the vehicle via network 1590), and / or the machine learning model can be used by server 1578 to remotely monitor the vehicle.

[0278] In some examples, server 1578 can receive data from vehicles and apply that data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 1578 may include a deep learning supercomputer powered by GPU 1584 and / or a dedicated AI computer, such as the DGX and DGX station machines developed by NVIDIA. However, in some examples, server 1578 may include a deep learning infrastructure in a data center that uses only CPU power.

[0279] The deep learning infrastructure of server 1578 may be capable of rapid real-time inference and can be used to assess and verify the health status of the processor, software, and / or associated hardware in vehicle 1500. For example, the deep learning infrastructure may receive periodic updates from vehicle 1500, such as image sequences and / or objects located in those image sequences by vehicle 1500 (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 1500. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 1500 has malfunctioned, then server 1578 may transmit a signal to vehicle 1500 instructing the vehicle's fail-safe computer to take control, notify passengers, and complete a safe stopping operation.

[0280] For inference, server 1578 may include GPU 1584 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0281] Example computing device

[0282] Figure 16 The following is a block diagram suitable for implementing some embodiments of the present disclosure: an example computing device 1600. The computing device 1600 may include an interconnect system 1602 that is directly or indirectly coupled to the following devices: a memory 1604, one or more central processing units (CPUs) 1606, one or more graphics processing units (GPUs) 1608, a communication interface 1610, I / O ports 1612, input / output components 1614, a power supply 1616, one or more presentation components 1618 (e.g., displays), and one or more logic units 1620.

[0283] although Figure 16 The various blocks are shown connected via an interconnect system 1602 with wiring, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 1618, such as a display device, may be considered an I / O component 1614 (e.g., if the display is a touchscreen). As another example, the CPU 1606 and / or GPU 1608 may include memory (e.g., memory 1604 may represent a storage device other than the memory of the GPU 1608, CPU 1606, and / or other components). In other words, Figure 16The computing devices mentioned are merely illustrative. No distinction is made between categories such as "workstation," "server," "laptop," "desktop," "tablet," "client device," "mobile device," "handheld device," "game console," "electronic control unit (ECU)," "virtual reality system," "augmented reality system," and / or other device or system types, as all of these are considered within the same category. Figure 16 Within the scope of computing devices.

[0284] Interconnect system 1602 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1602 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Fast (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. For example, CPU 1606 may be directly connected to memory 1604. Furthermore, CPU 1606 may be directly connected to GPU 1608. Where there is a direct or point-to-point connection between components, interconnect system 1602 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required in computing device 1600.

[0285] The memory 1604 may include any of a wide variety of computer-readable media. Computer-readable media can be any available medium that can be accessed by the computing device 1600. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. For example and without limitation, computer-readable media may include computer storage media and communication media.

[0286] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media, implemented in any way or by any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1604 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by computing device 1600. As used herein, computer storage media does not include the signal itself.

[0287] Computer storage media may include computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium. The term "modulated data signal" may refer to a signal whose characteristics are set or altered in a manner that encodes information into that signal. For example and without limitation, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as sound, RF, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0288] CPU 1606 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1600 to perform one or more of the methods and / or processes described herein. Each of CPU 1606 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a large number of software threads simultaneously. CPU 1606 may include any type of processor and may include different types of processors depending on the type of computing device 1600 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1600, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors such as math coprocessors, computing device 1600 may also include one or more CPUs 1606.

[0289] In addition to or as a replacement for CPU 1606, one or more GPUs 1608 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1600 to perform one or more of the methods and / or processes described herein. One or more GPUs 1608 may be integrated GPUs (e.g., having one or more CPUs 1606) and / or one or more GPUs 1608 may be discrete GPUs. In embodiments, one or more GPUs 1608 may be coprocessors of one or more CPUs 1606. Computing device 1600 may use GPUs 1608 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, one or more GPUs 1608 may be used for general-purpose computing on a GPU (GPGPU). One or more GPUs 1608 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPUs 1608 may generate pixel data for outputting an image in response to rendering commands (e.g., rendering commands received from CPU 1606 via a host interface). GPU 1608 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 1604. One or more GPUs 1608 may include two or more GPUs operating in parallel (e.g., via a link). The link may be directly connected to the GPUs (e.g., using NVLINK) or connected via a switch (e.g., using NVSwitch). When combined, each GPU 1608 may generate pixel data or GPGPU data for different portions of the output or different outputs (e.g., the first GPU for the first image, the second GPU for the second image). Each GPU may include its own memory or may share memory with other GPUs.

[0290] In addition to or as an alternative to CPU 1606 and / or GPU 1608, logic unit 1620 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1600 to perform one or more of the methods and / or processes described herein. In embodiments, CPU 1606, GPU 1608, and / or logic unit 1620 may execute any combination of methods, processes, and / or portions thereof, discretely or jointly. One or more logic units 1620 may be part of and / or integrated into one or more of CPU 1606 and / or GPU 1608, and / or one or more logic units 1620 may be discrete components or otherwise separate from CPU 1606 and / or GPU 1608. In embodiments, one or more logic units 1620 may be coprocessors of one or more CPUs 1606 and / or one or more GPUs 1608.

[0291] Examples of logic unit 1620 include one or more processing cores and / or components thereof, such as tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree traversal unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), I / O element, peripheral component interconnect (PCI) or peripheral component interconnect fast (PCIe) element, etc.

[0292] The communication interface 1610 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1600 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. The communication interface 1610 may include components and functions that enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication via Ethernet or InfiniBand), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0293] I / O port 1612 enables computing device 1600 to be logically coupled to other devices, including I / O component 1614, presentation component 1618, and / or other components, some of which may be built into (e.g., integrated into) computing device 1600. Illustrative I / O component 1614 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, and so on. I / O component 1614 can provide a Natural User Interface (NUI) for processing user-generated air gestures, voice, or other physiological input. In some instances, the input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, on-screen and adjacent-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of computing device 1600 (described in more detail below). Computing device 1600 may include depth cameras such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof for gesture detection and recognition. In addition, the computing device 1600 may include an accelerometer or gyroscope that enables motion detection (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 1600 to render immersive augmented reality or virtual reality.

[0294] Power supply 1616 may include hard-wired power supply, battery power supply, or a combination thereof. Power supply 1616 may supply power to computing device 1600 so that components of computing device 1600 can operate.

[0295] The presentation component 1618 may include a display (such as a monitor, touch screen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1618 may receive data from other components (such as GPU 1608, CPU 1606, etc.) and output that data (such as as images, videos, sounds, etc.).

[0296] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0297] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" could include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" could include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0298] Furthermore, the use of the term "based on" should not be interpreted as "only based on" or "based only on". Rather, "based on" the second element includes cases where the first element is based solely on the second element or on the second element and one or more additional elements.

[0299] This document describes in detail the subject matter of this disclosure to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have envisioned that the claimed subject matter may also be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms "step" and / or "block" may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A keyframe-based compression method, comprising: Based at least on map data and at a first frequency, determine multiple keyframes corresponding to a world model representing one or more areas around the vehicle; as well as Based at least on a specific keyframe among the plurality of keyframes and a plurality of positioning results determined relative to the vehicle, a plurality of world model frames corresponding to the world model are generated at a second frequency greater than the first frequency, wherein each world model frame generating the plurality of world model frames includes, at least based on each positioning result of the plurality of positioning results, spatially transforming the specific keyframe to the reference coordinate system of the vehicle.

2. The method of claim 1, wherein transforming the specific keyframe in space to the reference coordinate system comprises: The specific keyframe is spatially transformed to the reference coordinate system according to the second frequency and one or more spatial transformations determined over time.

3. The method of claim 1, wherein the plurality of keyframes are determined based on at least a plurality of lane maps.

4. The method according to claim 3, further comprising: The plurality of lane maps are generated based on at least one or more portions of the map data corresponding to multiple portions of a high-definition (HD) map, wherein the generation includes: incrementally increasing the set of valid lanes around the vehicle based on at least one or more portions of the map data.

5. The method of claim 4, wherein incrementally increasing the set of effective lanes around the vehicle for at least two sequential time frames out of a plurality of time frames comprises: At least one lane from a previous time frame is retained in the set, the at least one lane being kept within the horizontal line of the vehicle, the horizontal line being represented as the perimeter of a predefined area centered on the vehicle; Add each lane that is newly within the horizontal line to the set; as well as Remove from the set at least one lane that was retained in or added to the previous time frame that is no longer located within the horizontal line.

6. The method according to claim 4, further comprising: The first and second effective lane buffers are alternately filled with the currently increasing set of effective lanes, where: When the set of recently increasing effective lanes is stored in the second effective lane buffer, the first effective lane buffer is filled with the set of currently increasing effective lanes; and When the set of recently increasing effective lanes is stored in the first effective lane buffer, the second effective lane buffer is filled with the set of currently increasing effective lanes.

7. The method according to claim 1, further comprising: The first world model buffer and the second world model buffer are alternately populated with the plurality of keyframes, wherein: When the specific keyframe is stored in the second world model buffer, the first world model buffer is filled with the current keyframe from the plurality of keyframes currently being computed; and When the specific keyframe is stored in the first world model buffer, the second world model buffer is filled with the current keyframe.

8. The method of claim 1, wherein the plurality of keyframes are calculated at the first frequency instead of the second frequency, such that the computational compression ratio of the plurality of keyframes is at least based on the speed of the vehicle and is at least 10:

1.

9. The method of claim 1, wherein the plurality of keyframes are calculated at the first frequency instead of the second frequency, such that the computational compression ratio of the plurality of keyframes is at least based on the speed of the vehicle and is at least 40:

1.

10. A processing system based on keyframe compression, comprising: One or more processors are used to perform the following operations: Based at least on map data and at a first frequency, determine multiple keyframes corresponding to a world model representing one or more areas around the vehicle; as well as Based at least on a specific keyframe among the plurality of keyframes and a plurality of positioning results determined relative to the vehicle, a plurality of world model frames corresponding to the world model are generated at a second frequency greater than the first frequency, wherein each world model frame generating the plurality of world model frames includes, at least based on each positioning result of the plurality of positioning results, spatially transforming the specific keyframe to the reference coordinate system of the vehicle.

11. The system of claim 10, wherein transforming the specific keyframe in space to the reference coordinate system comprises: The specific keyframe is spatially transformed to the reference coordinate system according to the second frequency and one or more spatial transformations determined over time.

12. The system of claim 10, wherein the plurality of keyframes are determined based on at least a plurality of lane maps.

13. The system according to claim 12, further comprising: The plurality of lane maps are generated based on at least one or more portions of the map data corresponding to multiple portions of a high-definition (HD) map, wherein the generation includes: incrementally increasing the set of valid lanes around the vehicle based on at least one or more portions of the map data.

14. The system of claim 13, wherein incrementally increasing the set of effective lanes around the vehicle for at least two sequential time frames out of a plurality of time frames comprises: At least one lane from a previous time frame is retained in the set, the at least one lane being kept within the horizontal line of the vehicle, the horizontal line being represented as the perimeter of a predefined area centered on the vehicle; Add each lane that is newly within the horizontal line to the set; as well as Remove from the set at least one lane that was retained in or added to the previous time frame that is no longer located within the horizontal line.

15. The system according to claim 13, further comprising: The first and second effective lane buffers are alternately filled with the currently increasing set of effective lanes, where: When the set of recently increasing effective lanes is stored in the second effective lane buffer, the first effective lane buffer is filled with the set of currently increasing effective lanes; and When the set of recently increasing effective lanes is stored in the first effective lane buffer, the second effective lane buffer is filled with the set of currently increasing effective lanes.

16. The system of claim 10, further comprising: The first world model buffer and the second world model buffer are alternately populated with the plurality of keyframes, wherein: When the specific keyframe is stored in the second world model buffer, the first world model buffer is filled with the current keyframe from the plurality of keyframes currently being computed; and When the specific keyframe is stored in the first world model buffer, the second world model buffer is filled with the current keyframe.

17. The system of claim 10, wherein the plurality of keyframes are calculated at the first frequency instead of the second frequency, such that the computational compression ratio of the plurality of keyframes is at least based on the speed of the vehicle and is at least 10:

1.

18. The system of claim 10, wherein the plurality of keyframes are calculated at the first frequency instead of the second frequency, such that the computational compression ratio of the plurality of keyframes is at least based on the speed of the vehicle and is at least 40:

1.

19. A processor based on keyframe compression, comprising: The processing circuit is used to perform the following operations: Based at least on map data and at a first frequency, determine multiple keyframes corresponding to a world model representing one or more areas around the vehicle; as well as Based at least on a specific keyframe among the plurality of keyframes and a plurality of positioning results determined relative to the vehicle, a plurality of world model frames corresponding to the world model are generated at a second frequency greater than the first frequency, wherein each world model frame generating the plurality of world model frames includes, at least based on each positioning result of the plurality of positioning results, spatially transforming the specific keyframe to the reference coordinate system of the vehicle.

20. The processor of claim 19, further comprising: Multiple lane maps are generated based on at least one or more portions of the map data corresponding to multiple portions of a high-definition (HD) map, wherein: The generation includes incrementally increasing the set of effective lanes around the vehicle based on at least one or more portions of the map data; and The multiple keyframes are determined based on at least the multiple lane maps.

Images