Keyframe-based compression of world model representations in autonomous systems and applications

Abstract

To provide a world model system and keyframe-based compression in an application.SOLUTION: In various examples, a method includes a step of calculating a current keyframe. The current keyframe represents an area around an autonomous vehicle at current time based on map data. The method includes a step of generating a first world model frame by converting a previous keyframe to a coordinate frame of the autonomous vehicle at a first time prior to completing calculation of the current keyframe. The method includes a step of generating a second world model frame by converting the previous keyframe to the coordinate frame of the autonomous vehicle at a second time after the first time and prior to completion of the calculation of the current keyframe.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure generally relates to a world model of self-driving vehicles, and more particularly to compression of world model data based on key frames.

Background Art

[0002] Autonomous and semi-autonomous vehicles, also known as self-driving cars, driverless cars, highly automated driver assistance vehicles, or robotic vehicles, typically have the functionality to navigate them from a source location to a destination location without the need for a human driver to continuously monitor, control, and / or directly operate the vehicle. Automating driving is difficult for several reasons. For example, autonomous vehicles make driving decisions opportunistically using sensors, but vehicle sensors do not always have a clear view of everything. Vehicle sensors can be obscured or blocked by corners, gentle hills, and other vehicles, glare, objects, or road users (such as pedestrians, animals, cyclists). Input from vehicle sensors may not be processed early enough or quickly enough for autonomous planning or control functionality to make appropriate decisions. In addition, lanes and signs may be missing, knocked down, or hidden by foliage on the road, and thus may not be detectable by sensors. Furthermore, right-of-way road signs may not be easily visible to determine where a vehicle may be coming from, or to pull out of or exit a lane when there is a stopped obstacle that must be passed in an emergency or when passing is not allowed.

[0003] Autonomous vehicles can use accurate map data to mitigate some of the processing required to determine or confirm some of the aforementioned information in real time, instead of relying strictly on sensor data, or in addition to it. For example, an autonomous vehicle can use map data to construct a world model aligned with the vehicle's coordinate frame. For safe navigation of an autonomous vehicle, world model frames are typically calculated at a rate of about 30 world model frames per second. World model frames with large data sizes can consume considerable communication, processing, and / or storage resources in an autonomous vehicle. [Overview of the project] [Means for solving the problem]

[0004] The embodiments of this disclosure relate to keyframe-based compression of world model (WM) systems and applications.

[0005] In an exemplary embodiment, the method includes the step of calculating a current keyframe representing the area around the autonomous vehicle at the current time based on map data. The method also includes the step of generating a first WM frame by converting a preceding keyframe to a coordinate frame of the autonomous vehicle in a first time period prior to the completion of the calculation of the current keyframe. The method also includes the step of generating a second WM frame by converting a preceding keyframe to a coordinate frame of the autonomous vehicle in a second time period after the first time period but prior to the completion of the calculation of the current keyframe.

[0006] In another exemplary embodiment, the method includes the step of calculating, at a first frequency, a plurality of keyframes representing one or more areas around an autonomous vehicle based on map data. The method also includes the step of generating frames by converting the most recent keyframe to a coordinate frame of the autonomous vehicle at a second frequency greater than the first frequency, such that a plurality of WM frames are generated from at least one keyframe.

[0007] In another exemplary embodiment, the system includes one or more processors to implement first and second WM buffers, a WM producer node, and a WM transformer node. The WM producer node is connected to each of the first and second WM buffers to compute keyframes representing one or more areas around the autonomous vehicle at a first frequency based on map data, and to alternately feed the keyframes into the first and second WM buffers. The WM transformer node is connected to each of the first and second WM buffers to transform the most recent keyframe stored in the first or second WM buffer into a coordinate frame of the autonomous vehicle at a second frequency greater than the first frequency over time.

[0008] The system and method for WM keyframe-based compression are described in detail below with reference to the attached drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an exemplary overall system environment of an HD map system capable of interacting with multiple vehicles, according to one or more embodiments of the present disclosure. [Figure 2] This figure shows an exemplary system architecture of a vehicle computing system that may be included in the system environment of Figure 1, according to one or more embodiments of the present disclosure. [Figure 3] This figure shows an example of various layers of instructions in the HD map API of the vehicle calculation system of Figure 1, according to one or more embodiments of the present disclosure. [Figure 4] This figure shows an exemplary system architecture of an online HD map system that may be included in the system environment of Figure 1, according to one or more embodiments of the present disclosure. [Figure 5] This figure shows exemplary components of an HD map according to one or more embodiments of the present disclosure. [Figure 6A]This figure shows exemplary geographical regions that may be defined in an HD map according to one or more embodiments of the present disclosure. [Figure 6B] This figure shows exemplary geographical regions that may be defined in an HD map according to one or more embodiments of the present disclosure. [Figure 7] This figure shows an exemplary lane representation in an HD map according to one or more embodiments of the present disclosure. [Figure 8A] This figure shows exemplary lane elements and relationships between lane elements in an HD map according to one or more embodiments of the present disclosure. [Figure 8B] This figure shows exemplary lane elements and relationships between lane elements in an HD map according to one or more embodiments of the present disclosure. [Figure 9A] This figure shows an exemplary architecture for calculating a WM frame according to one or more embodiments of the present disclosure. [Figure 9B] This flowchart illustrates how one or more embodiments of the present disclosure may be implemented by the lane collector, segment graph producer, and / or other components of the architecture shown in Figure 9A. [Figure 9C] This figure shows a method for generating a WM frame according to one or more embodiments of the present disclosure. [Figure 10] This figure shows an exemplary LaneGraph calculation from a simple HD map according to one or more embodiments of the present disclosure. [Figure 11A] This figure shows an exemplary LaneGraph calculation from the HD map of a moving autonomous vehicle or rig according to one or more embodiments of the present disclosure. [Figure 11B] This figure shows an exemplary LaneGraph calculation from the HD map of a moving autonomous vehicle or rig according to one or more embodiments of the present disclosure. [Figure 12] This figure shows an exemplary implementation of a WM generator having three nodes, which may be included in the architecture of Figure 9A, according to one or more embodiments of the present disclosure. [Figure 13]A flowchart showing a method for generating a WM frame according to one or more embodiments of the present disclosure. [Figure 14] A flowchart showing another method for generating a WM frame according to one or more embodiments of the present disclosure. [Figure 15A] An illustration of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 15B] An example of the camera positions and fields of view of the exemplary autonomous vehicle of FIG. 15A according to some embodiments of the present disclosure. [Figure 15C] A block diagram of an exemplary system architecture of the exemplary autonomous vehicle of FIG. 15A according to some embodiments of the present disclosure. [Figure 15D] A system diagram of communication between a cloud-based server and the exemplary autonomous vehicle of FIG. 15A according to some embodiments of the present disclosure. [Figure 16] A block diagram of an exemplary computing device suitable for use in the implementation of some embodiments of the present disclosure.

Mode for Carrying Out the Invention

[0010] A system and method are disclosed for processing map data having a large data size in an autonomous vehicle. In one or more embodiments, the map data can include lanes, traffic signs and signals, waiting conditions, and the like, and can be used in an autonomous vehicle to construct a world model around the autonomous vehicle. Generally, the static content of each map-based world model is content that changes relatively slowly and, as such, need not be generated for every frame. Accordingly, some embodiments divide the generation of data in the autonomous vehicle into multiple tasks where tasks having a higher latency are executed less frequently than tasks having a lower latency. For example, the generation of data can be divided into a lower frequency world model keyframe generation task having a higher latency and a higher frequency transformation task having a lower latency. Alternatively or additionally, the tasks can include a higher frequency transformation calculation task having a lower latency. These tasks can repeat or iterate over time.

[0011] In an exemplary embodiment, world model keyframes around an autonomous vehicle are calculated over time and at a first frequency based on map data such as a lane graph. The most recent world model keyframe at any given time is converted over time and at a second frequency greater than the first frequency to coordinate frames of the autonomous vehicle in order to generate multiple world model frames, such that multiple world model frames are generated from each world model keyframe. In another exemplary embodiment, the current world model keyframe around an autonomous vehicle is calculated based on map data. A preceding world model keyframe may be converted to coordinate frames of the autonomous vehicle in a first time before the calculation of the current world model keyframe is completed in order to generate a first world model frame. The preceding world model keyframe may also be converted to coordinate frames of the autonomous vehicle in a second time after the first time and before the calculation of the current world model keyframe is completed in order to generate a second world model frame. In both exemplary embodiments, the higher latency calculation of world model keyframes is performed at a lower frequency than the lower latency generation of world model frames.

[0012] Some embodiments may implement a dual buffer for world model keyframes. For example, world model keyframes may be alternately loaded into a first world model buffer or a second world model buffer when they are calculated. While the current world model keyframe is being calculated and loaded into the first or second world model buffer, preceding or most recently calculated world model keyframes in the other of the second or first world model buffer are repeatedly converted to coordinate frames of the autonomous vehicle in order to generate a world model frame. The compression ratio of some embodiments described herein may depend on the speed of the autonomous vehicle. For example, the compression ratio may be at least 10:1 (e.g., 10:1, 12:1, 20:1, or any other ratio where the first number is at least 10 times the second number) when the autonomous vehicle is traveling at approximately 144.8 kilometers (90 miles) per hour (mph) or approximately 40 meters per second (m / s). Alternatively or in addition, the compression ratio may be at least 40:1 (for example, 40:1, 45:1, 50:1, or any other ratio where the first number is at least 40 times the second number) when the autonomous vehicle is traveling at approximately 22 mph or approximately 10 m / s.

[0013] Figure 1 shows an exemplary overall system environment of an HD map system 100 capable of interacting with multiple vehicles, according to one or more embodiments of the present disclosure. The HD map system 100 may comprise an online HD map system 110 capable of interacting with two or more vehicles 150 (e.g., vehicles 150A to 150D) of the HD map system 100. The vehicles 150 may be autonomous vehicles, semi-autonomous vehicles, or non-autonomous vehicles. Exemplary embodiments of the vehicles 150 are illustrated and described with respect to Figures 15A to 15D. It should be understood that these and other configurations described herein are described merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be excluded together. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0014] Continuing to refer to Figure 1, the online HD map system 110 may be configured to receive sensor data that can be captured by the vehicle sensors 105 (e.g., 105A-105D) of the vehicle 150, and to combine the data received from the vehicle 150 to generate and maintain an HD map. The online HD map system 110 may be configured to send the HD map data to the vehicle 150 for use when driving the vehicle 150. In some embodiments, the online HD map system 110 may be implemented as a cloud-based service that allows clients, such as a distributed computing system, for example, a vehicle computing system 120 (e.g., vehicle computing systems 120A-120D), to make requests for information and services. For example, the vehicle computing system 120 may make a request for HD map data for driving along a route, and the online HD map system 110 may provide the requested HD map data to the vehicle computing system 120. The vehicle computing system 120 may implement world model keyframe-based compression with respect to the HD map data received from the online HD map system 110.

[0015] Figure 1 and other figures use similar reference numerals to identify similar elements. The letters following a reference numeral, such as "105A," indicate that the text specifically refers to the element that has that particular reference numeral. Reference numerals in text that do not have following letters, such as "105," refer to one or all of the elements in the figure that have that reference numeral (for example, "105" in the text refers to the reference numerals "105A" and / or "105N" in the figure).

[0016] The online HD map system 110 may comprise a vehicle interface module 160 and an HD map store 165. The online HD map system 110 may be configured to interact with a vehicle calculation system 120 of various vehicles 150 using the vehicle interface module 160. The online HD map system 110 may be configured to store map information for various geographical areas in the HD map store 165. The online HD map system 110 may be configured to include modules other than those shown in Figure 1, for example, various other modules as shown in Figure 4 and further described herein.

[0017] In this disclosure, a module may include code and routines configured to enable a corresponding system (e.g., a corresponding computing system) to perform one or more of the operations described therein. In addition or otherwise, any given module may be implemented using hardware including any number of processors, microprocessors (e.g., for performing or controlling the performance of one or more operations), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or any suitable combination of two or more thereof. In alternative or additionally, any given module may be implemented using a combination of hardware and software. In this disclosure, operations described as being performed by a module may include operations that the module can instruct the corresponding system to perform.

[0018] Furthermore, the distinctions and separations of different modules shown in this disclosure are for the purpose of aiding in the description of the operations performed and are not intended to limit them. For example, depending on the implementation, operations described in relation to two or more of the modules described in this disclosure may be performed by what could be considered the same module. Furthermore, one or more operations of a module may be divided among what could be considered one or more other modules or submodules, depending on the implementation.

[0019] The online HD map system 110 may be configured to receive sensor data collected by sensors of two or more vehicles 150, for example, hundreds or thousands of automobiles. The sensor data may include any data that can be obtained by vehicle sensors that may be relevant to the generation of the HD map. For example, the sensor data may include LIDAR data, captured images, etc. In addition or by alternative means, the sensor data may include information that can describe the current state of the vehicle 150, such as the vehicle 150's position and motion parameters.

[0020] Vehicle 150 may be configured to provide sensor data 115 that can be captured while driving along various routes and to transmit it to an online HD map system 110. The online HD map system 110 may be configured to use the sensor data 115 received from vehicle 150 to create and update an HD map that describes the areas that vehicle 150 may be driving through. The online HD map system 110 may be configured to build a high-resolution map based on the aggregate sensor data 115 that can be received from vehicle 150 and to store the HD map information in an HD map store 165. The foregoing describes one method for generating an HD map. More generally, the HD maps described herein may be generated using the foregoing and / or other suitable methods.

[0021] The online HD map system 110 may be configured to send HD map data 125 to the vehicle 150 upon request from the vehicle 150.

[0022] For example, in an instance where a particular vehicle 150 is scheduled to drive along a route, the specific vehicle calculation system 120 of the particular vehicle 150 may be configured to provide the online HD map system 110 with information describing the route being traveled. In response, the online HD map system 110 may be configured to provide HD map data 125 of an HD map (for example, representing an area containing the route) related to the route, which may facilitate navigation and driving along the route by the particular vehicle 150.

[0023] In one embodiment, the online HD map system 110 may be configured to send portions of the HD map data to the vehicle 150 in a compressed format so that the transmitted data can consume less bandwidth. The online HD map system 110 may be configured to receive information from various vehicles 150 that describes HD map data that can be stored in the vehicle 150's local HD map store (for example, the local HD map store 275 in Figure 2).

[0024] In some embodiments, the online HD map system 110 may determine that a particular vehicle 150 may not have a particular portion of the HD map data locally stored in the local HD map store of the vehicle calculation system 120 of that particular vehicle 150. In these or other embodiments, in response to such determination, the online HD map system 110 may be configured to send the portion of the HD map data to the vehicle 150.

[0025] In some embodiments, the online HD map system 110 may determine that a particular vehicle 150 may have previously received HD map data relating to the same geographical area as a particular portion of HD map data. In these or other embodiments, the online HD map system 110 may determine that a particular portion of HD map data may be an updated version of previously received HD map data updated by the online HD map system 110 since the particular vehicle 150 last received the previous HD map data. In some embodiments, the online HD map system 110 may transmit updates to that portion of HD map data that can be stored in a particular vehicle 150. This may allow the online HD map system 110 to reduce or minimize the amount of HD map data that can be communicated with the vehicle 150, and further to retain locally stored HD map data in regularly updated vehicles.

[0026] The vehicle 150 may include vehicle sensors 105 (e.g., vehicle sensors 105A to D), vehicle control devices 130 (e.g., vehicle control devices 130A to 130D), and vehicle computing systems 120 (e.g., vehicle computer systems 120A to 120D). The vehicle sensors 105 may be configured to detect the surroundings of the vehicle 150. In these or other embodiments, the vehicle sensors 105 may detect information describing the current state of the vehicle 150, such as information describing the position and motion parameters of the vehicle 150.

[0027] The vehicle sensor 105 may include a camera, a light detection and ranging sensor (LIDAR), a Global Navigation Satellite System (GNSS) receiver, such as a Global Positioning System (GPS) navigation system, an inertial measurement unit (IMU), and others. The vehicle sensor 105 may include one or more cameras capable of capturing images of the vehicle's surroundings. The LIDAR may survey the vehicle's surroundings by illuminating its target with laser pulses and measuring the distance to the target by measuring the reflected pulses. The GPS navigation system may determine the vehicle 150's position based on signals from satellites. The IMU may include an electronic device configured to measure and report motion data of the vehicle 150, such as velocity, acceleration, direction of motion, speed, and angular velocity, using an accelerometer and a gyroscope or a combination of other measuring instruments.

[0028] The vehicle control device 130 may be configured to control the physical motion of the vehicle 150, such as acceleration, direction changes, starting, stopping, etc. The vehicle control device 130 may include machinery for controlling the accelerator, brakes, steering wheel, etc. The vehicle calculation system 120 can periodically and / or continuously provide control signals to the vehicle control device 130, causing the vehicle 150 to drive along a selected route.

[0029] The vehicle calculation system 120 may be configured to perform a variety of tasks, including processing data collected by sensors and map data received from the online HD map system 110. The vehicle calculation system 120 may also be configured to process data for sending to the online HD map system 110. An example of the vehicle calculation system 120 is further illustrated in Figure 2, which will be described in more detail.

[0030] Interaction between the vehicle calculation system 120 and the online HD map system 110 may be performed over a network, for example, over the Internet. The network may be configured to enable communication between the vehicle calculation system 120 and the online HD map system 110. In some embodiments, the network may be configured to use standard communication technologies and / or protocols. Data exchanged over the network may be represented using technologies and / or formats including Hypertext Markup Language (HTML), Extended Markup Language (XML), etc. In addition, all or some of the links may be encrypted using conventional cryptographic technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec). In some embodiments, entities may use custom and / or proprietary data communication technologies.

[0031] Vehicle calculation system Figure 2 shows an exemplary system architecture of a vehicle calculation system 120 according to one or more embodiments of the present disclosure. The vehicle calculation system 120 may include a cognitive module 210, a prediction module 215, a planning module 220, a control module 225, a local HD map store 275, an HD map system interface 280, a map inconsistency module 290, and an HD map application programming interface (API) 205. Various modules of the vehicle calculation system 120 may be configured to process various types of data, including sensor data 230, behavior models 235, routes 240, and physical constraints 245. In some embodiments, the vehicle calculation system 120 may include more or fewer modules. Functionality described as being implemented by a particular module may be implemented by other modules.

[0032] Referring to Figures 2 and 1, in some embodiments, the vehicle calculation system 120 may include a perception module 210. The perception module 210 may be configured to receive sensor data 230 from vehicle sensors 105 of the vehicle 150. The sensor data 230 may include data collected by the vehicle's cameras, LIDAR, IMU, GPS navigation system, etc. The perception module 210 may also be configured to use the sensor data 230 to determine what objects are around the corresponding vehicle 150, details of the road the corresponding vehicle 150 is traveling on, etc. In addition, the perception module 210 may be configured to process the sensor data 230 and input it into a data structure that stores the sensor data 230 and to provide information or commands to the prediction module 215 of the vehicle calculation system 120.

[0033] The prediction module 215 may be configured to interpret data provided by the recognition module 210 using a behavior model (235) of recognized objects to determine whether an object is likely to move or is about to move. For example, the prediction module 215 may determine that objects identified as vehicles, people, etc., may be moving or likely to move, while objects representing road signs may not be moving. The prediction module 215 may also be configured to use the behavior model 235 of various types of objects to determine whether they are likely to move. In addition, the prediction module 215 may be configured to provide predictions of various objects to the planning module 200 of the vehicle calculation system 120 in order to plan subsequent actions that the corresponding vehicle 150 may take next.

[0034] The planning module 220 may be configured to receive information from the prediction module 215 describing the surroundings of the corresponding vehicle 150, and a route 240 which may indicate or determine the destination of the vehicle 150 and indicate possible paths the vehicle 150 can take to reach the destination.

[0035] The planning module 220 may also be configured to use information from the prediction module 215 and the route 240 to plan a sequence of actions that the vehicle 150 may take within a short time interval, for example, within the next few seconds. In some embodiments, the planning module 220 may be configured to specify the sequence of actions as one or more points representing nearby locations that the vehicle 150 may next drive through. The planning module 220 may be configured to provide the control module 225 with details of the plan, including the sequence of actions that will be taken by the corresponding vehicle 150. The plan may indicate whether the corresponding vehicle 150 may perform one or more subsequent actions, such as changing lanes, making a U-turn, accelerating or decelerating by increasing speed, etc.

[0036] The control module 225 may be configured to determine control signals that can be sent to the vehicle control device 130 of the corresponding vehicle 150 based on a plan that may be received from the planning module 220. For example, if the corresponding vehicle 150 is currently at point A and the plan specifies that the corresponding vehicle 150 should next proceed to nearby point B, the control module 225 can determine control signals for the vehicle control device 130 that will allow the corresponding vehicle 150 to proceed from point A to point B in a safe and smooth manner, for example, without taking a sharp turn or a zigzag path from point A to point B. The path that the corresponding vehicle 150 may take to proceed from point A to point B may depend on the current speed and direction of the corresponding vehicle 150 and the position of point B relative to point A. For example, if the current speed of the corresponding vehicle 150 is high, the corresponding vehicle 150 may be able to make a wider turn compared to another vehicle driving slowly.

[0037] The control module 225 may also be configured to receive physical constraints 245 as input. The physical constraints 245 may include the physical capabilities of the corresponding vehicle 150. For example, a corresponding vehicle 150 of a particular type and model may be able to safely perform certain types of vehicle maneuvers, such as acceleration and turning, which another vehicle of a different type and model may not be able to safely perform. In addition, the control module 225 may be configured to incorporate the physical constraints 245 when determining the control signals for the vehicle control device 130 of the corresponding vehicle 150. In addition, the control module 225 may be configured to send control signals to the vehicle control device 130 that can cause the vehicle 150 to perform a specified sequence of actions and move the corresponding vehicle 150 as planned according to a predetermined set of actions. In some embodiments, the steps described above may be continuously repeated every few seconds, allowing the corresponding vehicle 150 to be driven safely along a route that may be planned for the corresponding vehicle 150.

[0038] Various modules of the vehicle computing system 120, including the cognitive module 210, the prediction module 215, and the planning module 220, can be configured to receive map information and perform their respective calculations. The corresponding vehicle 150 may store HD map data in the local HD map store 275. Modules of the vehicle computing system 120 may interact with HD map data using the HD map application programming interface (API) 205.

[0039] The HD Map API 205 may provide one or more application programming interfaces (APIs) that can be called by modules for accessing map information. The HD Map System Interface 280 may be configured to allow the vehicle calculation system 120 to interact with the online HD Map System 110 over a network (not shown). The local HD Map Store 275 may store map data in a format that may be specified by the online HD Map System 110. The HD Map API 205 may be configured to have the ability to process map data formats as provided by the online HD Map System 110. The HD Map API 205 may be configured to provide the vehicle calculation system 120 with an interface for interacting with HD map data. The HD Map API 205 may include several APIs, including the Localization API 250, the Landmark Map API 255, the Route API 270, the 3D Map API 265, and the Map Update API 285.

[0040] The localization API 250 may be configured to determine the current location of the corresponding vehicle 150, for example, the location of the corresponding vehicle 150 along a given route. The localization API 250 may be configured to include a localized API for determining the location of the corresponding vehicle 150 within an HD map and within a certain level of precision. The vehicle calculation system 120 may be configured to use its location as an accurate relative location (for example, within a certain level of precision) for making other queries, such as feature queries, navigable spatial queries, and occupancy map queries, which are further described herein.

[0041] The localization API 250 may be configured to receive inputs including one or more of the following: position provided by GPS, vehicle motion data provided by an IMU, LiDAR scanner data, camera images, etc. The localization API 250 may be configured to return the precise position of the corresponding vehicle 150 as latitude and longitude coordinates. The coordinates that can be returned by the localization API 250 may be more accurate than the GPS coordinates used as input, for example, the output of the localization API 250 may have a distance measurement accuracy of 2 to 30 cm. In some embodiments, the vehicle calculation system 120 may be configured to call the localization API 250 to periodically determine the position of the corresponding vehicle 150 based on LiDAR using scanner data, for example, at a frequency of 10 hertz (Hz).

[0042] The vehicle calculation system 120 may be configured to call the localization API 250 to determine the vehicle position at a higher rate (e.g., 60 Hz) if GPS or IMU data is available at that rate. In addition, the vehicle calculation system 120 may be configured to store a location history record as internal state to improve the accuracy of subsequent localization calls. The location history record may store a history of location from points in time, such as when the corresponding vehicle 150 was switched off / stopped. The localization API 250 may include a route localization API which can be configured to generate an accurate (e.g., within a specified precision) route specifying lanes based on an HD map. The route localization API may be configured to receive a route from source to destination as input via one or more third-party maps and may be configured to generate a highly accurate (e.g., within a specified precision, e.g., within 30 cm) route represented as a connected graph of navigable lanes along the input route based on an HD map.

[0043] The Landmark Map API 255 may be configured to provide a geometric and semantic description of the world around a corresponding vehicle 150, for example, a description of the different parts of the lane that the corresponding vehicle 150 is currently traveling in. The Landmark Map API 255 may include APIs that can be configured to enable queries based on the landmark map, for example, a Lane Fetch API and a Feature Fetch API. The Lane Fetch API may be configured to provide lane information and a Feature Fetch API related to the corresponding vehicle 150. The Lane Fetch API may also be configured to receive a location, for example, the location of the corresponding vehicle 150 specified using latitude and longitude, as input, and return lane information related to the input location. In addition, the Lane Fetch API may be configured to specify a distance parameter indicating the distance to the input location from which the lane information can be retrieved. Furthermore, the Feature Fetch API may be configured to receive information identifying one or more lane elements and return landmark features related to the specified lane elements. Landmark features may include a spatial description, which may be specific to that type of landmark, for each landmark.

[0044] The 3D map API 265 may be configured to provide a spatial three-dimensional (3D) representation of the road and access to various physical objects around the road as stored in the local HD map store 275. The 3D map API 265 may include a navigable surface fetch API and an occupancy grid fetch API. The navigable surface fetch API may be configured to receive identifiers of one or more lane elements as input and return the navigable boundary lines of the specified lane elements. The occupancy grid fetch API may also be configured to receive locations as input, for example, the latitude and longitude of a corresponding vehicle 150, and return information describing the occupancy of the road surface and all objects available in the HD map near its location. The information describing the occupancy may include a hierarchical volume grid of some or all locations that are considered to be occupied in the HD map. The occupancy grid may include information at a high resolution near navigable areas, for example, curves and ridges, and at a relatively low resolution in less important areas, for example, trees and walls beyond a curve. In addition, the fetch API for occupied grids can be configured to be useful for detecting obstacles and changing direction as needed.

[0045] The 3D map API 265 also includes map update APIs, such as a map update download API and a map update upload API. The map update download API may be configured to receive a planned route identifier as input and download data related to all planned routes or map updates for a specific planned route. The map update upload API may be configured to upload data collected by the vehicle calculation system 120 to the online HD map system 110. The map update upload API may allow the online HD map system 110 to retain updated HD map data stored in the online HD map system 110 based on changes in map data that can be observed by vehicle sensors 105 of vehicles 150 driving along various routes.

[0046] The Route API 270 may be configured to return route information, including the complete route and parts of the route between the source and destination, as the corresponding vehicle 150 travels along the route. The 3D Map API 265 may be configured to enable querying of the online HD map system 110 or HD maps. The Route API 270 may include an Add Planned Route API and a Get Planned Route API. The Add Planned Route API may be configured to provide the online HD map system 110 with information describing a planned route, so that the information describing the associated HD map can be downloaded by the vehicle calculation system 120 and kept up to date. The Add Planned Route API may be configured to receive, as input, a route specified using a polyline expressed with respect to latitude and longitude, and a time-to-live (TTL) parameter specifying a time period after which the route data can be deleted. Thus, the Add Planned Route API may be configured to allow the vehicle 150 to indicate a route that it plans to take as autonomous travel in the near future. The Add Planned Route API may be configured to align the route with the HD map, record the route and its TTL value, and ensure that the HD map data of the route stored in the vehicle calculation system 120 is updated (for example, to the latest version). The Get Planned Route API may be configured to return a list of planned routes and provide information describing the routes identified by route identifiers.

[0047] The map update API 285 may be configured to manage operations related to updating map data in both the local HD map store 275 and the HD map store 165 stored in the online HD map system 110. Thus, a module in the vehicle calculation system 120 may be configured to call the map update API 285 to download data from the online HD map system 110 to the vehicle calculation system 120 for storage in the local HD map store 275. The map update API 285 may also be configured to allow the vehicle calculation system 120 to determine whether information monitored by the vehicle sensor 105 indicates a discrepancy in the map information provided by the online HD map system 110, and to upload the data to the online HD map system 110, which may result in the online HD map system 110 updating the map data stored in the HD map store 165 provided to other vehicles 150.

[0048] The map inconsistency module 290 may be configured to work with the map update API 285 to determine map inconsistencies and to communicate map inconsistency information to the online HD map system 110. In some embodiments, determining a map inconsistency involves comparing sensor data 230 at a particular location with HD map data at that particular location. For example, HD map data may indicate that a highway lane should be available to vehicle 150, but sensor data 230 may indicate that there is construction work taking place in that lane that has been closed for use or is otherwise no longer available. When the map inconsistency module 290 detects a map inconsistency, the corresponding vehicle 150 sends an update message to the online HD map system 110 containing information about the detected map inconsistency. The map inconsistency module 290 may be configured to construct an update message that may include a vehicle identifier (ID), one or more time stamps, the route traveled, the lane element ID of the traversed lane element, the type of inconsistency, the magnitude of the inconsistency, an inconsistency fingerprint that helps identify a double inconsistency warning message, the size of the message, and so on. In some embodiments, one or more operations of the map inconsistency module 290 can be handled at least partially by the map data acquisition module 460 in Figure 4, as detailed below.

[0049] In some embodiments, the corresponding vehicle 150 may be configured to send an update message to the online HD map system 110 or the local HD map store 275 when it detects a map inconsistency, and / or to send update messages periodically. For example, the corresponding vehicle 150 may be configured to record inconsistencies and report them to the online HD map system 110 via an update message once every time interval (e.g., 30 minutes) or distance traveled (e.g., 16.09 kilometers (10 miles)). The online HD map system 110 may be configured to manage and prioritize update messages, as will be further described below with reference to the map data acquisition module 460.

[0050] In some embodiments, the corresponding vehicle 150 may be configured to send an update message to the online HD map system 110 only when it reaches or docks with a high-bandwidth access point. After the corresponding vehicle 150 is connected to the Internet (e.g., a network), the corresponding vehicle 150 may be configured to send either a matched update message or a set of update messages. In one or more embodiments, illustrative, but not limited, messages may include update messages constructed since the last high-bandwidth access point was reached or docked. The use of high-bandwidth access points may be useful for transmitting large amounts of data. In some embodiments, upon receiving an acknowledgment message that a matched update message or one or more update messages have been received by the online HD map system 110, the corresponding vehicle 150 marks the data to be deleted and schedules a local deletion process and / or deletes the data. Alternatively, the corresponding vehicle 150 may report to the online HD map system 110 periodically based on time, for example, every hour.

[0051] The map inconsistency module 290 may be configured to function in response to messages from the online HD map system 110 and to perform operations related to inconsistency identification. For example, when it receives a message requesting data about a specific location along the route of a corresponding vehicle 150, the map inconsistency module 290 may be configured to instruct one or more vehicle sensors 105 of the corresponding vehicle 150 to collect that data and report it to the map inconsistency module 290. Upon receiving the data, the map inconsistency module 290 may be configured to immediately construct a message containing the data, such as at the next scheduled time in a periodic schedule or at the next high-bandwidth access point, and send the message to the online HD map system 110.

[0052] The map inconsistency module 290 may be configured to determine the urgency of a determined map inconsistency that will be included in any update to an HD map containing the inconsistent area. For example, there may be two urgency levels: low urgency and high urgency. The online HD map system 110 may consider the urgency of an update message when deciding how to process the information in the update message, as detailed below with respect to the map data acquisition module 460. For example, a single lane closure on a remote or rural back road may be determined to have low urgency, while a complete closure of a major highway in a city of a million people may be determined to have high urgency. In some instances, high urgency update messages may be processed by the online HD map system 110 before low urgency update messages.

[0053] In some embodiments, the corresponding vehicle 150 may be configured to continuously record sensor data 230 to the online HD map system 110 and encode the relevant portion of it for message generation in response to a request for additional data at a particular location, for example. In one embodiment, the vehicle 150 may be configured to simply delete the continuously recorded sensor data 230 after receiving confirmation from the online HD map system 110 that none of the sensor data 230 is required by the online HD map system 110.

[0054] Figure 3 shows an example of various layers of instructions in the HD map API 205 of the vehicle computing system 120 according to one or more embodiments of the present disclosure. Different vehicle manufacturers may have different procedures or instructions for receiving information from vehicle sensors 105 and for controlling the vehicle control device 130. Furthermore, different vendors may offer different computer platforms having autonomous driving capabilities, such as the collection and analysis of vehicle sensor data. Examples of computer platforms for autonomous vehicles include platforms offered by vendors such as NVIDIA Corporation, QUALCOMM, and INTEL. These platforms may provide functionality for use by autonomous vehicle manufacturers in the manufacture of autonomous vehicles 150. Vehicle manufacturers may use any one or more computer platforms for autonomous vehicles 150.

[0055] The online HD map system 110 may be configured to provide a library for processing HD maps based on instructions specific to the vehicle manufacturer and instructions specific to the vehicle dealership platform. The library can provide access to HD map data and enable the vehicle 150 to interact with the online HD map system 110.

[0056] As shown in Figure 3, the HD Map API 205 may be implemented as a library including a vehicle manufacturer adapter 310, a computer platform adapter 320, and a common HD Map API layer 330. The common HD Map API layer 330 may be implemented to include comprehensive instructions that can be used across two or more vehicle computing platforms and vehicle manufacturers. The computer platform adapter 320 may be implemented to include instructions that may be specific to each computer platform. For example, the common HD Map API layer 330 may be implemented to call the computer platform adapter 320 to receive data from sensors supported by a particular computer platform. The vehicle manufacturer adapter 310 may be implemented to include instructions specific to the vehicle manufacturer. For example, the common HD Map API layer 330 may be implemented to call functionality provided by the vehicle manufacturer adapter 310 to send specific control instructions to the vehicle control unit 130.

[0057] The online HD map system 110 may be implemented to store computer platform adapters 320 for two or more computer platforms and vehicle manufacturer adapters 310 for two or more vehicle manufacturers. The online HD map system 110 may be implemented to determine a specific vehicle manufacturer and a specific computer platform for a particular autonomous vehicle 150. The online HD map system 110 may be implemented to select a vehicle manufacturer adapter 310 for a specific vehicle manufacturer and a computer platform adapter 320 for a specific computer platform for that particular vehicle 150. In addition, the online HD map system 110 may be implemented to send instructions for the selected vehicle manufacturer adapters 310 and selected computer platform adapters 320 to the vehicle computing system 120 of that particular autonomous vehicle 150. The vehicle computing system 120 of that particular autonomous vehicle 150 may be implemented to install the received vehicle manufacturer adapters 310 and computing platform adapters 320. The vehicle calculation system 120 may be implemented to periodically check or verify whether the online HD map system 110 has updates for the installed vehicle manufacturer adapter 310 and calculation platform adapter 320. Additionally, if a more recent update is available compared to the version installed on the vehicle 150, the vehicle calculation system 120 may be implemented to request, receive, and install the latest update.

[0058] HD Map System Architecture Figure 4 shows an exemplary system architecture of an online HD map system 110 according to one or more embodiments of the present disclosure. The online HD map system 110 may include a map creation module 410, a map update module 420, a map data encoding module 430, a load balancing module 440, a map accuracy management module 450, a vehicle interface module 160, a map data acquisition module 460, and an HD map store 165. Some embodiments of the online HD map system 110 may include more or fewer modules than those shown in Figure 4. Functionality shown as being performed by a particular module may be implemented by other modules. In some embodiments, the online HD map system 110 may be implemented as a distributed system including two or more processing systems.

[0059] The map creation module 410 may be configured to create HD map data for an HD map from sensor data collected from several vehicles (e.g., 150A-150D) driving along various routes. The map update module 420 may be configured to update previously calculated HD map data by receiving more recent information (e.g., sensor data) from vehicles 150 that have recently traveled along routes where the map information has changed. For example, if certain road signs have changed, or lane information has changed as a result of construction in the area, the map update module 420 may be configured to update the HD map and the corresponding HD map data accordingly. The map data encoding module 430 may be configured to efficiently store the data (e.g., compress the HD map data) and encode the HD map data so that it can be sent to the vehicles 150. The load balancing module 440 may be configured to maintain load balance across vehicles 150 so that requests for data from vehicles 150 are distributed across different vehicles 150 (for example, evenly distributed) (for example, the load distribution between different vehicles 150 is within each other's threshold amounts). The map accuracy management module 450 may be configured to maintain relatively high accuracy of HD map data using various techniques, even though the information received from individual vehicles may not have the same accuracy.

[0060] In some embodiments, the map data acquisition module 460 may be configured to monitor the vehicles 150 and process status updates from the vehicles 150 to determine whether to request additional data related to one or more specific locations from one or more types of vehicles 150.

[0061] Figure 5 shows exemplary components of an HD map 510 according to one or more embodiments of the present disclosure. The HD map 510 may be configured to include HD map data for maps of several geographical areas. In the present disclosure, a reference to a map or HD map, for example, HD map 510, may include a reference to map data corresponding to such a map. Furthermore, a reference to information for each map may also include a reference to map data for that map.

[0062] In some embodiments, the HD map 510 of a geographical area may include a landmark map (LMap) 520 and an occupancy map (OMap) 530. The landmark map 520 may include information or representations of driving routes (e.g., lanes, yield lines, safely navigable spaces, private roads, unpaved roads, etc.), pedestrian routes (e.g., crosswalks, sidewalks, etc.), and landmark objects (e.g., road signs, buildings, etc.). For example, the landmark map 520 may include information describing lanes, including the spatial location of the lanes and semantic information about each lane. The spatial location of the lanes may include geometric locations in latitude, longitude, and altitude with high precision, e.g., within 30 cm or better. The semantic information of the lanes may include constraints such as direction, speed, lane type (e.g., lane for going straight, left-turn lane, right-turn lane, exit lane, etc.), constraints regarding left crossings, and connectivity to other lanes.

[0063] In some embodiments, the landmark map 520 may include information describing road signs, including stop lines, yield lines, the spatial locations of pedestrian crossings, safely navigable spaces, the spatial locations of speed bumps, curves, and the spatial locations of all types of signage related to driving constraints. Examples of road signs described in the HD map 510 may include traffic signs, stop signs, traffic signals, speed limits, one-way signs, no entry signs, and yield signs (vehicles, pedestrians, animals).

[0064] In some embodiments, the information contained in the landmark map 520 may be related to a confidence value that measures the probability that the representation is accurate. The representation of an object is accurate when the information describing the object matches the object's attributes (e.g., driving path, pedestrian path, landmark object, etc.). For example, the representation of a driving path may be considered accurate when its spatial position and semantic information can match the attributes of the driving path (e.g., physical measurements, constraints, etc.). The vehicle calculation system 120 (e.g., the planning module 220) can use the confidence value to control the vehicle 150. For example, if the representation of a landmark object is associated with a high confidence value in the landmark map 520, but the vehicle 150 does not detect the landmark object based on both the vehicle sensor 105 and corresponding observations of the environment around the vehicle 150, the vehicle calculation system 120 may be configured to control the vehicle 150 to avoid the landmark object estimated to be present based on the high confidence value, or to control the vehicle 150 to comply with driving constraints imposed by the landmark object (for example, to make the vehicle 150 yield based on yield signs on the landmark map).

[0065] In some embodiments, the occupancy map 530 may include a spatial 3D representation of roads and physical objects around roads. The occupancy map 530 may also be referred to herein as an occupancy grid. Similarly, the data stored in the occupancy map 530 may also be referred to herein as occupancy grid data. The 3D representation of roads and physical objects around roads may be associated with confidence scores indicating the likelihood of objects being present at their locations. The occupancy map 530 may be represented in several other ways. In some embodiments, the occupancy map 530 may be represented as a 3D mesh geometry (collection of triangles) that can encompass surfaces. In some embodiments, the occupancy map 530 may be represented as a collection of 3D points that can encompass surfaces. In some embodiments, the occupancy map 530 may be represented using a 3D volume grid of cells with a resolution of 5-10 cm. Each cell may indicate whether a surface exists in that cell and, if so, the orientation along which the surface may be oriented.

[0066] The occupied map 530 can take up a large amount of memory compared to the landmark map 520. For example, 1GB / mile of data can take up 4x10 15 It may be used by the occupied map 530, which provides a US map (including 6.437 million kilometers (4 million miles) of roads) occupying 10 bytes or 4 petabytes. Thus, the online HD map system 110 and the vehicle calculation system 120 may be configured to store and transfer map data using data compression techniques, thereby reducing storage and transmission costs. Thus, the techniques disclosed herein may help improve the autonomous driving of autonomous vehicles by improving the efficiency of data storage and transmission with respect to autonomous driving operations and capabilities.

[0067] In some embodiments, the HD map 510 may use or not rely on data that would normally be included in a map, such as addresses, street names, the ability to geocode addresses, and the ability to calculate routes between place names or addresses. The vehicle calculation system 120 or the online HD map system 110 can access other map systems, such as (and without limitation) OpenStreetMap, to obtain this information. Thus, the vehicle calculation system 120 or the online HD map system 110 can receive map information from tools such as OpenStreetMap and convert the information into routes based on the HD map 510, or convert the information so that it may be compatible with us on the HD map 510.

[0068] Geographical areas within HD maps The online HD map system 110 may be configured to divide a large physical area into geographic regions and to store a representation of each geographic region. Each geographic region may represent adjacent areas bounded by a geometric shape—for example (and without limitation), a rectangle or a square. In some embodiments, the online HD map system 110 may be configured to divide a physical area into geographic regions of similar size, regardless of the amount of data required to store a representation of each geographic region. In some embodiments, the online HD map system 110 can divide a physical area into geographic regions of different sizes, where the size of each geographic region may be determined based on the amount of information required to represent the geographic region. For example, a geographic region representing a densely populated area with many streets may represent a smaller physical area than a geographic region representing a sparsely populated area with very few streets. In some embodiments, the online HD map system 110 may be configured to determine the size of geographic regions based on an estimate of the amount of information that can be used to store various elements of a physical area associated with an HD map 510.

[0069] In one embodiment, the online HD map system 110 may represent geographical areas using objects or data records that may include (without limitation) various attributes, including: a unique identifier for the geographical area, a unique name for the geographical area, a description of the geographical area's boundaries using bounding boxes corresponding to a set of latitude and longitude coordinates, and a collection of landmark features and occupied grid data.

[0070] Figures 6A–6B show exemplary geographical regions 610A and 610B that may be defined in an HD map according to one or more embodiments of the present disclosure. Figure 6A shows a square geographical region 610A. Figure 6B shows two adjacent geographical regions 610A and 610B. The online HD map system 110 may be configured to store data in a representation of geographical regions that can enable a smooth transition from one geographical region to another when a vehicle 150 is driving across geographical region boundaries.

[0071] In some embodiments, as shown in Figures 6A-6B, each geographical region may include a buffer of a predetermined width (area) around it. The buffer may include redundant map data around one or more or all sides of the geographical region (for example, if the geographical region is bounded by a rectangle). Thus, in some embodiments, if the geographical region may have a particular shape, the geographical region may be bounded by a buffer that may be a larger version of that shape. As an example, Figure 6A shows the boundary line 620 of a buffer approximately 50 meters around geographical region 610A and the boundary line 630 of a buffer 100 meters around geographical region 610A.

[0072] In some embodiments, the vehicle calculation system 120 may be configured to switch the current geographical region of a corresponding vehicle 150 from one geographical region to an adjacent geographical region when the corresponding vehicle 150 crosses a predetermined (e.g., defined) threshold distance within a buffer. For example, as shown in Figure 6B, the corresponding vehicle 150 starts at position 650A within geographical region 610A. The corresponding vehicle 150 may traverse along the route and arrive at position 650B, where it crosses the boundary of geographical region 610 but remains within the buffer boundary 620. Thus, the vehicle calculation system 120 for the corresponding vehicle 150 may continue to use geographical region 610A as the vehicle 150's current geographical region. After the corresponding vehicle 150 crosses the buffer boundary 620 at position 650C, the vehicle calculation system 120 may be configured to switch the current geographical region of the corresponding vehicle 150 from geographical region 610A to geographical region 610B. The use of buffers can reduce or prevent rapid changes in the current geographical area of ​​vehicle 150 as a result of vehicle 150 traveling along a route that can closely track the boundaries of the geographical area.

[0073] Lane representation in HD map The HD map system 100 may represent lane information for streets within the HD map. While the described embodiments may refer to streets, the technique may be applicable to any highway, side street, main street, boulevard, route, etc., on which a vehicle can travel. The HD map system 100 can use lanes as a reference frame for routing purposes and for the localization of vehicles 150. Lanes represented by the HD map system 100 may include explicitly marked lanes, e.g., white and yellow striped lanes; lanes that may be implicit, e.g., those that may have two directions of travel despite being on a rural road without lines or curves; and implicit routes that may function as lanes, e.g., routes that a turning vehicle may create when entering a lane from another lane.

[0074] The HD map system 100 can store lane-related information, such as landmark features like road signs and traffic signals associated with the lane, lane-related occupancy grids for obstacle detection, and lane-related navigable spaces, so that in an emergency when the vehicle 150 makes an unplanned movement out of its lane, the vehicle 150 can plan / react. Thus, the HD map system 100 can store a representation of the lane network to enable the vehicle 150 to plan a legal route between its source and destination and to add a reference frame for the vehicle 150's real-time sensing and control. The HD map system 100 stores information and provides an API that enables the vehicle 150 to determine the lane in which the vehicle 150 is currently located, the exact position of the vehicle 150 in relation to the lane geometry, and any and all related features / data associated with the lane and adjacent and connected lanes.

[0075] Figure 7 shows an exemplary lane representation in an HD map according to one or more embodiments of the present disclosure. Figure 7 shows a vehicle 710 at a traffic intersection. The HD map system 100 provides the vehicle 710 with access to map data that may be relevant to the autonomous driving of the vehicle 710. This may include, for example, features 720A and 720B that may be related to lanes but may not be the features closest to the vehicle 710. Thus, the HD map system 100 may store lane-centered representations of the data that can represent the relationship between features and lanes, so that the vehicle 710 can efficiently extract features given lanes.

[0076] The HD map system 100 can provide an HD map representing a portion of a lane as a lane element. A lane element can specify the lane boundary and various constraints, including the legal direction in which a vehicle 710 can travel within the lane element, the speed at which a vehicle can drive within the lane element, and whether the lane element may relate only to left turns or only to right turns. In some embodiments, the HD map system 100 can provide a map representing the lane element as a continuous geometric portion of a single vehicle lane. The HD map system 100 can store objects or data structures that can represent a lane element, including information such as the geometric boundary of the lane, the direction of driving along the lane, vehicle constraints for driving within the lane, such as speed limits, relationships with connecting lanes including entering and exiting lanes, termination constraints, such as whether the lane ends at a stop line, yield sign, or speed bump, and relationships with road features related to autonomous driving, such as the location of traffic lights and road signs.

[0077] Examples of lane elements represented by the HD maps of the HD map system 100 may include, for example and without limitation, the following: a right lane on a highway, a lane on a road, a left-turn lane, a U-turn from a left-turn lane to another lane, a merging lane from an on-ramp, an exit lane on an off-ramp, and a private road. The HD map system 100 may include an HD map representing a single-lane road using two lane elements, one for each direction. The HD map system 100 may represent a shared central U-turn lane similar to a single-lane road.

[0078] Figures 8A and 8B show exemplary lane elements (e.g., LaneEl) and relationships between lane elements in an HD map according to one or more embodiments of the present disclosure. Each of the lane elements in Figures 8A and 8B may include, be included in, or correspond to one or more MapsLane described elsewhere in this specification. Figure 8A shows an example of a T-junction on a road showing lane element 810A (e.g., an example of a straight LaneEl) which can be connected to lane element 810C (e.g., another straight LaneEl) via lane 810B (e.g., a curved LaneEl) and to lane 810E (e.g., another straight LaneEl) via a turning lane 810D (e.g., another curved LaneEl). Figure 8B shows an example of a Y-junction on a road having lane 810F which is directly connected to lane 810H and connected to lane 810I via lane 810G. The HD map system 100 can determine a route from a source location to a destination location as a sequence of connected lane elements that can be traversed to reach the destination location from the source location.

[0079] This document describes an exemplary architecture and method for constructing a LocalMap around an autonomous vehicle. In the following discussion: MapsLane refers to a lane defined by a polyline with lane attributes; LaneSegment refers to a mutually exclusive segment stitched from MapsLane; RoadSegment refers to an atomic element of the core layer of an HD map; LocalLayout refers to a section of an HD map in which the relative transformation of all RoadSegments is calculated with reference to the root RoadSegment; LaneGraph refers to a comprehensive list of all possible combinations of paths in LocalLayout; LaneGraph lanes refer to drivable paths in LocalLayout; and LaneInfo refers to the MapsLane intermediate metadata used for LaneGraph calculations. In this specification, the aforementioned and other specific terms may be generalized to refer to their corresponding comprehensive structures, concepts, or equivalents. For example, a "lane graph" could refer to a comprehensive list of all possible combinations of paths in a section of an HD map, where the relative transformations of all atomic elements in the core layer of the HD map are calculated by referencing the root atomic element, regardless of whether the section of the HD map is called LocalLayout or whether the atomic elements are called RoadSegment.

[0080] LocalMap can provide a local view of an HD map, which can be achieved by computing a comprehensive LaneGraph containing all possible combinations of paths in LocalLayout. During the process of computing the LaneGraph, other HD map features within LocalLayout, such as WaitGroups, TSLObjects, or similar, may be created.

[0081] In one or more embodiments, WaitGroups may include collections or groups of individual wait elements. In at least one embodiment, a wait element constitutes a singular or atomic representation of how a wait condition (potential yield scenario) is represented. Each wait element consists of an ego-competitor pair (e.g., consisting of an ego vehicle and at least one competing obstacle or object in the environment) and a wait state that describes the wait element's current priority. A wait group ("WaitGroup") associates multiple wait elements together. In one or more embodiments, wait elements are grouped together for wait conditions that are intended to be cleared together. For example, taking an unprotected left may be represented as being obligated to yield to pedestrians at a crosswalk at the end of a U-turn. These two groupings of wait elements make it clearer that an ego vehicle should wait to enter the ego lane of either wait element until both wait elements can be cleared.

[0082] A TSL (Traffic Sign Light) object encodes the 3D geometry of a traffic signal light / sign. In one or more embodiments, wait rules may be used to define the association between wait states and light classes, and may be used to resolve the current wait state applicable to a wait element. For example, typical wait rules for a wait element might look like "Proceed" for blue, "Stop at Entrance" for red, and "Take-way-translate" for yellow. In one or more embodiments, TSL objects and wait rules may be implemented using internal metadata, which may be used when processing the resolution of the wait state corresponding to the wait element.

[0083] Each lane in the LaneGraph may be aware of its other related lanes (merging, separating, and intersecting lanes), changing lanes (left and right lanes), and other map features (WaitGroups, TSLObjects, etc.). The LaneGraph and HD map features may reside within the coordinate system of an autonomous vehicle, which may be referred to separately or additionally as the rig coordinate system or ego coordinate system. The world model definition may be used as the output of the LocalMap to represent this data.

[0084] World Model Calculation Figure 9A shows an exemplary architecture 900 for calculating a world model (WM) frame according to one or more embodiments of the present disclosure. As shown, the architecture 900 may include a lane collector 902, a segment graph producer 904, a lane graph producer 906, and a WM generator 908. Generally, the lane collector 902 can collect and hold MapsLane in LocalLayout, the segment graph producer 904 can stitch MapsLane into mutually exclusive LaneSegments to produce a coarse map (or SegmentGraph), the lane graph producer 906 can assemble all combinations of lanes from LaneSegments, each lane may have its own unique tracking identifier (ID) that can be consistent across frames, and the WM generator 908 can input WM structures of lanes, waiting conditions, or similar, such as WM keyframes, and can convert the calculated LaneGraph into the Ego coordinate system to determine the ego lane. The output of the WM generator 908 may be referred to as a WM frame. The lane collector 902, segment graph producer 904, lane graph producer 906, and WM generator 908 will be discussed in more detail below.

[0085] The lane collector 902 can extract the collection of lanes from the LocalLayout. Each lane in the LocalLayout can be stored in a wrapper structure. In the exemplary embodiment, the wrapper structure for each lane in the LocalLayout is referred to as LocalLayoutLane.

[0086] Lane collector 902 may have a LaneBuffer, which it will put into the current LocalLayout build.

[0087] Lane collector 902 can collect lanes by iterating over all road segments in the current LocalLayout build.

[0088] A LocalLayout for LaneGraph calculations can be constructed for a default radius around a sensor suite (also referred to as a "rig") of sensors typically positioned along the perimeter of an autonomous vehicle. The default radius around the rig can define a circular area with the rig at its center. This circular area may be referred to as the LocalLayout, and / or the boundary or perimeter of the circular area may be referred to as the horizon of the autonomous vehicle or rig.

[0089] The segment graph producer 904 can construct a coarse map made up of LocalMapSegments. MapsLane in LocalLayout can be stitched into mutually exclusive LocalMapSegments at any given time such that the MapsLane is associated with only one LocalMapSegment. A LocalMapSegment can recognize its other related LocalMapSegments (incoming, outgoing, intersecting, and changing lane segments). A LocalMapSegment may also be an atomic unit of the coarse map, also referred to herein as a SegmentGraph.

[0090] MapsLanes that participate in a LocalMapSegment may be encoded in a MapsLane structure. The MapsLane structure may include the IDs and / or offsets of the participating MapsLanes in the LocalMapSegment (this information may be used to deduce lane changes and other lane assignment attributes). A MapsLane may have left and / or right lane changes and / or other information that may be stored in a MapLaneChange structure. The MapLaneChange structure may be used to compute a SegmentChange structure.

[0091] Connection information can be encoded in the SegmentConnection structure.

[0092] Lane change information can be encoded in a SegmentChange structure.

[0093] Intersecting segments can be encoded in the SegmentContentions structure.

[0094] Waiting condition information can be stored in the SegmentWaitCondition structure.

[0095] Figure 9B is a flowchart showing Method 910, which may be implemented by the lane collector 902, segment graph producer 904, and / or other components of the architecture 900 of Figure 9A, according to some embodiments of the present disclosure. Each block of Method 910 and / or other methods described herein includes a computation process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. The method may also be performed as computer-available instructions stored in a computer storage medium. The method may be provided, to name a few, by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or by plugging into another product. In some implementations, Method 910 and / or other methods described herein are stored in and / or performed by a vehicle computing system, for example, a vehicle computing system 120 described elsewhere herein.

[0096] Generally, method 910 may include a segment graph producer 904 consuming MapsLane and / or LocalLayoutLane collected by a lane collector 902 and stitching them into a list of unique LocalMapSegments to produce a rough map or SegmentGraph.

[0097] More specifically, method 910 includes selecting a seed lane in block 912. In an exemplary implementation, the seed lane is selected by analyzing the lanes in the rig's horizon (e.g., MapsLane, LocalLayoutLane) until a lane that does not contain an incoming lane is identified, and then selecting the identified lane as the seed lane. More generally, the seed lane may be selected in block 912 according to any appropriate criteria.

[0098] Method 910 includes, in block 914, collecting connected lanes in the reverse direction. In an exemplary implementation, block 914 includes collecting lanes in the horizon of a rig that is transiently connected to a seed lane in the reverse direction until a stopping condition is met.

[0099] Method 910 includes collecting connected lanes in the forward direction in block 916. In an exemplary implementation, block 916 includes collecting lanes in the horizon of the rig that are transiently connected to the seed lane in the forward direction until a stopping condition is met. The stopping condition in block 916 may be the same as or different from the stopping condition in block 914. Alternatively or additionally, the stopping conditions for lane collection in the reverse and / or forward directions may include one or more of the following: - MapsLane and / or LocalLayoutLane participate in a junction route. A junction route may include a junction of one MapsLane or LocalLayoutLane with another MapsLane or LocalLayoutLane (such as an ego lane and a connecting merging lane). Each MapsLane and / or LocalLayoutLane participating in a junction route belongs to the corresponding LaneSegment generated for the associated junction route. - MapsLane and / or LocalLayoutLane connections have multiple connecting lanes. - The direction of operation for connected MapsLane and / or LocalLayoutLane is bidirectional. - The map connecting the lanes is not within the current LocalLayout (for example, it is outside the rig's horizon).

[0100] Method 910 includes stitching the selected and collected MapsLane and / or LocalLayoutLane (for example, the seed lane selected in block 912 and the lanes collected in blocks 914 and 916) into a LaneSegment in block 918. In an exemplary implementation, the MapsLane and / or LocalLayoutLane are stitched according to the following convention: - MapsLane and / or LocalLayoutLane participating in the junction route are stitched into a LaneSegment, and metadata such as competitor segments, wait elements, and traffic rules are recorded. - Paths within a LocalLayout that have MapsLane and / or LocalLayoutLanes with zero incoming / outgoing paths are stitched into their respective LaneSegments. - Any unprocessed MapsLane and / or LocalLayoutLane left within the lane buffer are stitched into their respective LaneSegments.

[0101] After MapsLane and / or LocalLayoutLane are stitched into LaneSegment in block 918, metadata such as segment relationship information, segment change information, or similar may be collected in several implementations.

[0102] In at least one embodiment, the segment graph producer 904 may implement or facilitate implementations of one or more of the following functions: The updateSegmentGraph function may consume a valid LocalLayout build, which may be sequentially supplied to the lane collector 902 to collect LocalLayoutLanes, which may be used to construct a SegmentGraph. In some embodiments, the updateSegmentGraph function is implemented in the public API of the segment graph producer 904. The computeSegmentGraph function may consume a LaneBuffer fed by the lane collector 902 to construct a SegmentGraph. The computeSegmentGraph function may be a private function of the segment graph producer 904 and / or may be called from the updateSegmentGraph function. The computeSegment function may compute a new unique segment in the current LocalLayout, the input LocalLayoutLane may be used as a seed lane, which may be extended in both the forward and reverse directions until a stopping condition is met, as described elsewhere herein. The computeSegment function may be a private function of the segment graph producer 904 and / or may be called from the computeSegmentGraph function. The collectSegmentLanes function can collect the MapLane (defined by StitchDirection) of a segment in a given direction, which has the relevant data stored in the MapsLane structure, and the MapLane of a segment is collected from a pool of MapLanes in LaneBuffer (e.g., the output of the lane collector 902). For a given segment, the collectSegmentLanes function may be called twice from the computeSegment function to extend the segment in both the reverse and forward directions. The updateSegmentGraphInfo function may be called from the updateSegmentGraph function after the SegmentGraph has been computed.The updateSegmentGraphInfo function can help collect metadata by iterating over all segments in the SegmentGraph. The getSegmentConnections function can collect segment connections in the SegmentGraph. The getSegmentConnections function may be private and / or can be called from the updateSegmentGraphInfo function. The getSegmentChanges function can collect all segment changes in the SegmentGraph. The getSegmentChanges function may be private and / or can be called from the updateSegmentGraphInfo function.

[0103] The LaneGraph Producer 906 can assemble a comprehensive LaneGraph by performing a depth-first lookup on the LocalMapSegment graph. In some implementations, the updateLaneGraph function can be implemented in the public API of the LaneGraph Producer 906. The updateLaneGraph function can update the SegmentGraph in a new LocalLayout build by calling the updateSegmentGraph function and calculating the decomposed LaneGraph lanes by performing a depth-first lookup on the new SegmentGraph. Thus, the LaneGraph Producer 906 can generate a LaneGraph containing decomposed LaneGraph lanes and associated metadata, each decomposed LaneGraph lane having a unique, stable ID for tracking LaneGraph lanes between frames. The decomposed LaneGraph can be used, for example, by the WM Generator 908 to produce output structures such as WM frames. Ego lane hints can be determined based on the rig's position in the LaneGraph geometry. The LaneGraph lane ID can be used with a position query to track stable ego lanes.

[0104] The ID of a split lane can be calculated based on the participating MapsLane. An ID manager can be used to calculate a unique ID for a LaneGraph lane. The ID manager may maintain a hash table (or hash map) with a list of MapsLane IDs as keys and LaneGraph lane IDs as corresponding values. Thus, the IDs of the MapsLanes participating in a split lane can be used as keys to maintain consistent LaneGraph lane IDs across frames.

[0105] In some embodiments, WM keyframes add triplet point constraints to LaneChunks such that for points on the lane center of a LaneChunk, corresponding left and right divider points exist. However, this is not a constraint on HD maps. For stitched lane channels within a WM keyframe, triplets can be fabricated by projecting points onto each of the other two and three polylines converging with them from different directions, which can occur in loops of all three polylines within a lane channel.

[0106] In line with the foregoing, Figure 9C shows a method 920 for generating a WM frame according to some embodiments of the present disclosure. In 922, the vehicle calculation system 120 requests a LaneGraph from the lane graph producer 906. In 924, the lane graph producer updates the LocalLayout. In 926, the lane graph producer requests a SegmentGraph from the segment graph producer 904. In 928, the segment graph producer 904 requests lanes in the LocalLayout from the lane collector 902. In 930, the lane collector 902 collects lanes in the LocalLayout and feeds them into the LaneBuffer 932. In 934, the segment graph producer 904 retrieves junction information from the LocalLayout. In 936, the segment graph producer 904 stitches the junction routes. In 938, the segment graph producer 904 stitches the collected lanes into mutually exclusive LaneSegments. In 940, the segment graph producer 904 retrieves metadata for the LaneSegments. In 942, the segment graph producer 904 returns the SegmentGraph (and associated metadata) to the lane graph producer 906. In 944, the lane graph producer calculates the decomposed LaneGraph. In 946, the lane graph producer 906 retrieves metadata for the LaneGraph lanes in the LaneGraph. In 948, the lane graph producer 906 returns the LaneGraph (and associated metadata) to the vehicle calculation system 120. In 950, the vehicle calculation system requests a WM frame from the WM generator 908. In 952, the WM generator 908 generates the WM frame. In step 954, the WM generator 908 returns the WM frame to the vehicle calculation system 120.

[0107] Figure 10 shows an exemplary LaneGraph calculation from a simple HD map 1002 according to some embodiments of the present disclosure. The HD map 1002 includes a RoadSegment having R from MapsLaneA. A lane collector 902 collects MapsLane in LocalLayout. In this example, it is assumed that each of R from MapsLaneA is located in LocalLayout and is collected by the lane collector 902.

[0108] The segment graph producer 904 may then stitch R from MapsLaneA to mutually exclusive LaneSegmentLS-A through LS-F to produce a SegmentGraph 1004. For example, MapsLaneA, B, and C may be stitched to LaneSegmentLS-A, MapsLaneG, H, and I may be stitched to LaneSegmentLS-B, and so on. The SegmentGraph 1004 and / or associated metadata may identify one or more of the previous LaneSegment, the next LaneSegment, and competing LaneSegments for each LaneSegment.

[0109] Finally, the lane graph producer 906 can assemble all combinations of lanes from LaneSegmentLS-A to LS-F to generate a LaneGraph 1006 containing all route combinations of LaneSegment as LaneGraph lanes LGL-A to LGL-E. The first LaneGraph lane LGL-A, which includes LaneSegmentLS-C, LS-A, and LS-D, is represented in Figure 10 as a short dashed line. The second LaneGraph lane LGL-B, which includes LaneSegmentLS-B, LS-A, and LS-D, is represented in Figure 10 as a dashed line. The third LaneGraph lane LGL-C, which includes LaneSegmentLS-C, LS-A, and LS-E, is represented in Figure 10 as a solid line. The fourth LaneGraph lane LGL-D, which includes LaneSegmentLS-B, LS-A, and LS-E, is represented in Figure 10 as a long dashed line. The fifth LaneGraph lane, LGL-E, which includes LaneSegmentLS-F, is represented in Figure 10 as a dashed line. In Figure 10 and other figures of LaneGraphs with overlapping portions, the overlapping portions of LaneGraph lanes are spatially canceled out from each other in a way that is visually perceptible. In practice, overlapping portions of LaneGraph lanes within a LaneGraph do not necessarily have to be spatially separated.

[0110] Figures 11A and 11B illustrate exemplary LaneGraph calculations from the HD map 1002 of Figure 10 for a moving autonomous vehicle or rig 1102 according to some embodiments of the present disclosure. When rig 1102 is at three different locations on the HD map 1002 as shown in Figure 11A, the illustrated LaneGraph calculations are performed at three different times or frames, labeled frame 0, frame 1, and frame 2. In frame 0, rig 1102 is located on MapsLane H, driving in the direction from MapsLane I to MapsLane G. In frame 1, rig 1102 is located on MapsLane B, driving in the direction from MapsLane C to MapsLane A. In frame 2, rig 1102 is located on MapsLane K, driving in the direction from MapsLane I to MapsLane G.

[0111] As shown in the figure, rig 1102 has a LocalLayout 1104 that is updated as rig 1102 progresses. In frame 0, LocalLayout 1104 includes MapsLane G, H, I, C, and D. In frame 1, LocalLayout 1104 includes MapsLane A, B, C, P, Q, and R. In frame 2, LocalLayout 1104 includes MapsLane A, O, J, K, and L.

[0112] In each frame, the lane collector 902 collects all MapsLane in LocalLayout 1104. In some implementations, as shown in Figure 11A, the lane collector 902 collects all MapsLane in LocalLayout by incrementally increasing the collection 1106 of valid lanes around rig 1102 in LocalLayout 1104. More specifically, in each frame, the lane collector 902 may hold in collection 1106 each MapsLane from preceding frames remaining in rig 1102's LocalLayout 1104 or horizon, add each newly present MapsLane in rig 1102's LocalLayout 1104 or horizon to collection 1106, and remove from collection 1106 each MapsLane held or added in preceding frames that are no longer in rig 1102's LocalLayout 1104 or horizon. For example, in frame 0—which has no preceding frame—MapsLane G, H, I, C, and D are each added to the collection 1106 of valid lanes. In another example, in frame 1, MapsLane C is retained in collection 1106, MapsLane A, B, C, P, Q, and R are newly located within the LocalLayout 1104 or horizon of rig 1102 and are therefore added to collection 1106, and MapsLane G, H, I, and D, which were included in the previous frame (frame 0), are removed from the collection 1106 of valid lanes, shown as invalid lanes 1108 in Figure 11A. As another example, in frame 2, MapsLaneA is retained in collection 1106, MapsLaneO, L, K, and J are newly located within LocalLayout 1104 or Horizon of rig 1102 and are therefore added to collection 1106, and MapsLaneB, C, P, Q, and R, which were included in the previous frame (frame 1), are removed from the collection 1106 of valid lanes, shown as invalid lanes 1108 in Figure 11A.

[0113] In some implementations, LaneBuffer may be implemented as a dual buffer containing a first and second active lane buffer for collecting active lanes. For example, the first active lane buffer may receive the current incrementally augmented collection of active lanes between frames, while the second active lane buffer stores the preceding incrementally augmented collection of active lanes. The roles of the first and second active lane buffers can be stitched together so that in the next frame, the second active lane buffer may receive a new current incrementally augmented collection of active lanes between frames, while the first active lane buffer stores the preceding incrementally augmented collection of active lanes (e.g., the current incrementally augmented collection before the active lane). The incremental augmentation of active lane collection and the use of a dual active lane buffer may allow the generation of a SegmentGraph and / or LaneGraph from the preceding incrementally augmented collection of active lanes while simultaneously incrementally augmenting the current collection of active lanes.

[0114] Figure 11B shows the SegmentGraph generated in each frame. For example, in frame 0, the segment graph producer 904 can generate the SegmentGraph shown in frame 0 by stitching MapsLaneG, H, and I to LaneSegment 1106, MapsLaneC to LaneSegment 1108, and MapsLaneD to LaneSegment 1110. In frame 1, the segment graph producer 904 can generate the SegmentGraph shown in frame 1 by stitching MapsLaneA, B, and C to LaneSegment 1112 and MapsLaneP, Q, and R to LaneSegment 1114. In frame 2, the segment graph producer 904 may stitch MapsLaneA to LaneSegment 1116, MapsLaneO to LaneSegment 1118, and MapsLaneJ, K, and L to LaneSegment 1120 to generate the SegmentGraph shown in frame 2. Although not shown in Figure 11B, the SegmentGraph and / or associated metadata in each frame may identify one or more of the previous LaneSegment, the next LaneSegment, and competing LaneSegments for each LaneSegment.

[0115] Figure 11B also shows the LaneGraph generated in each frame. For example, in frame 0, the lane graph producer 906 may assemble all combinations of lanes from Lane Segments 1106, 1108, and 1110 to generate the LaneGraph shown in frame 0, which includes all route combinations of Lane Segments 1106, 1108, and 1110 as LaneGraph lanes 1122 and 1124. In frame 1, the lane graph producer 906 may assemble all combinations of lanes from Lane Segments 1112 and 1114 to generate the LaneGraph shown in frame 1, which includes all route combinations of Lane Segments 1112 and 1114 as LaneGraph lanes 1126 and 1128. In frame 2, the lane graph producer 906 may assemble all combinations of lanes from Lane Segments 1116, 1118, and 1120 to generate a LaneGraph shown in frame 2 that includes all route combinations of Lane Segments 1116, 1118, and 1120 as LaneGraph lanes 1130 and 1132.

[0116] Figure 11B further includes a legend 1134 showing different types of LaneGraph lanes that may be included in one or more frames of the LaneGraph. The LaneGraph lane currently being driven in by the vehicle or machine corresponding to rig 1102 is referred to as the “ego lane.” LaneGraph lanes 1122, 1126, and 1132 are examples of ego lanes. A LaneGraph lane merging with an ego lane is referred to as a “merging lane.” LaneGraph lane 1124 is an example of a merging lane. A LaneGraph lane separating from an ego lane is referred to as a “separating lane.” LaneGraph lane 1130 is an example of a separating lane. A LaneGraph lane intersecting with an ego lane is referred to as a “competitor lane.” LaneGraph lane 1128 is an example of a competitor lane.

[0117] For the safe navigation of autonomous vehicles, WM frames are typically computed at a rate of approximately 30 WM frames / second. Large data-size WM frames can consume significant communication, processing, and / or storage resources in autonomous vehicles. Therefore, some embodiments of this specification divide the generation of data for autonomous vehicles, e.g., WM frames, into multiple tasks, where tasks with higher latency are performed less frequently than tasks with lower latency. For example, data generation may be divided into a high-latency, low-frequency WM keyframe generation task and a low-latency, high-frequency conversion task. Alternatively or additionally, the tasks may include a low-latency, high-frequency conversion calculation task. These tasks can be repeated or iterated over time. For example, WM keyframes may be computed at a first frequency, while other WM frames (e.g., non-keyframes) may be computed at a second frequency higher than the first frequency.

[0118] In this and other implementations, the WM generator 908 may be divided into two, three, or any other number of nodes, where each node handles one of the tasks according to a corresponding frequency. Figure 12 shows an exemplary implementation of the WM generator 908 having three nodes 1202, 1204, and 1206 according to some embodiments of the present disclosure. The three nodes 1202, 1204, and 1206 include, in particular, a WM producer 1202, a WM attitude calculation node 1204, and a WM transformer 1206.

[0119] Generally, the WM generator 908 may receive a LaneGraph 1208 and localization results 1210 as inputs and output WM frames 1212 to one or more WM consumers 1213. Each localization result 1210 may identify the current position of the autonomous vehicle in the current WM frame 1212. The WM consumer 1213 may include, for example, a vehicle computing system 120 that can use or consume the WM frame 1212 to control the operation of the corresponding autonomous vehicle 150 or rig. Alternatively or additionally, the WM consumer 1213 may include any other system, device, application, or similar that uses or consumes the WM frame 1212.

[0120] WM Producer 1202 may be a node with high latency (compared to nodes 1204 and 1206) and low frequency (compared to nodes 1204 and 1206), while WM Attitude Calculation Node 1204 and WM Transformer 1206 may be nodes with low latency (compared to node 1202) and high frequency (compared to node 1202).

[0121] In general, the WM producer 1202 can calculate WM keyframes 1214 from the LaneGraph 1208. The calculation of WM keyframes may be a relatively high-latency calculation that can be performed by the WM producer 1202 at a first frequency, for example, about 10 Hz. According to some implementations, the WM producer 1202 implements a dual buffer for the calculation of WM keyframes 1214, in which it alternately feeds the current WM keyframe into one of the two buffers 1216, 1218, while a preceding WM keyframe 1214, for example, the immediately preceding one, is stored in one of the two buffers 1216, 1218. Buffers 1216, 1218 may optionally be referred to as world model buffers to distinguish them from other buffers described herein. Buffers 1216 and 1218, which store the preceding WM keyframe 1214 at any given time, can be read by the WM pose calculation node 1204 and the WM transformer node 1206 during the performance of their respective tasks. When the calculation of the current WM keyframe is complete, the current WM keyframe becomes the preceding WM keyframe 1214 in the corresponding one of the buffers 1216 and 1218, and a new current WM keyframe can be calculated by feeding the new current WM keyframe into the other of the two buffers 1216 and 1218. In this scheme, each of the buffers 1216 and 1218 can alternately store the preceding WM keyframe 1214 or the current WM keyframe can be fed into the buffer while the current WM keyframe is being calculated.

[0122] The WM attitude calculation node 1204 may calculate a valid transformation 1220 for each WM frame's preceding WM keyframe 1214 at a second frequency and from the localization result 1210 and the preceding WM keyframe 1214. The localization result 1210 may include the rig's current position at any given time and for any given WM frame generated by the WM generator 908. The rig's current position may be used by the WM attitude calculation node 1204 to determine the corresponding valid transformation 1220 (e.g., a local map to rig transformation matrix) which will transform the preceding WM keyframe 1214 into the rig's coordinate frame. The second frequency may be greater than the first frequency so that multiple valid transformations 1220 can be calculated sequentially for a given preceding WM keyframe 1214. In one example where the first frequency is 10 Hz and the second frequency is 30 Hz, the WM attitude calculation node 1204 can calculate three consecutive effective transformations 1220 for each preceding WM keyframe 1214.

[0123] The WM transformer 1206 may transform a preceding WM keyframe 1214 into a rig coordinate frame using a valid transform 1220. For example, the WM transformer 1206 may sequentially transform a preceding WM keyframe 1214 into a rig coordinate frame using a sequential valid transform 1220 to generate a sequential WM frame 1212. The WM transformer 1206 may perform the transform at a second frequency such that any given preceding WM keyframe 1214 can be transformed sequentially multiple times to generate multiple sequential WM frames 1212.

[0124] Figure 13 is a flowchart illustrating Method 1300 for generating WM frames according to some embodiments of the present disclosure. Each block of Method 1300 and / or other methods described herein includes a computation process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. The method may also be performed as computer-available instructions stored on a computer storage medium. The method may be provided, to name a few, by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or by plugging into another product. In some implementations, Method 1300 and / or other methods described herein are stored in and / or performed by a vehicle computing system, for example, a vehicle computing system 120 as described elsewhere herein. For example, Method 1300 may be performed at least in part by a WM generator 908 in a vehicle computing system 120 of an autonomous vehicle 150 or rig.

[0125] Method 1300 includes, in block 1302, calculating the current WM keyframe around the autonomous vehicle based on map data. For example, block 1302 may include the WM producer 1202 of the WM generator 908 calculating the current WM keyframe from the current LaneGraph 1208.

[0126] Method 1300 includes, in block 1304, converting a preceding WM keyframe to an autonomous vehicle coordinate frame in a first time period before completing the calculation of the current WM keyframe in order to generate a first WM frame. For example, block 1304 may include the WM transformer 1206 of the WM generator 908 converting a preceding WM keyframe 1214 to an autonomous vehicle coordinate frame in a first time period while the calculation of the current WM keyframe is being completed and in the preceding first time period to generate a first WM frame 1212.

[0127] Method 1300 includes, in block 1306, converting a preceding WM keyframe to an autonomous vehicle coordinate frame to generate a second WM frame in a second time after a first time and before the calculation of the current WM keyframe is completed. For example, block 1306 may include the WM transformer 1206 of the WM generator 908 converting the same preceding WM keyframe 1214 to an autonomous vehicle coordinate frame to generate a second WM frame 1212 in a second time after a first time and before the completion of the calculation of the current WM keyframe.

[0128] In some implementations, method 1300 may further include determining a transformation to be applied to a preceding WM keyframe. In this example, transforming a preceding WM keyframe into an autonomous vehicle coordinate frame may include transforming a preceding WM keyframe into an autonomous vehicle coordinate frame according to the determined transformation. For example, the WM attitude calculation node 1204 of the WM generator 908 may determine, for example, calculate, a valid transformation 1220 for the preceding WM keyframe 1214 based on the localization result 1214, and the preceding WM keyframe 1214 may be transformed into an autonomous vehicle coordinate frame according to the valid transformation 1220.

[0129] The map data on which the current WM keyframe is calculated may include, for example, a lane graph such as one of the LaneGraphs described herein. Thus, calculating the current WM keyframe around the autonomous vehicle based on the map data may include calculating the current WM keyframe around the autonomous vehicle based on the lane graph. Some embodiments of Method 1300 may further include generating a lane graph from a portion of an HD map, which includes incrementally increasing the collection of active lanes around the autonomous vehicle from a portion of the HD map, as described with respect to Figure 11A, for example. For example, the incrementally increased collection of active lanes may be stored in a first active lane buffer, and Method 1300 may further include feeding the current incrementally increased collection of active lanes into a second active lane buffer.

[0130] Alternatively or additionally, preceding WM keyframes may be stored in a first WM buffer, and method 1300 may further include loading the current WM keyframe into a second WM buffer when the current WM keyframe is calculated. As an example, the first WM buffer may include buffer 1216 in Figure 12, and the second WM buffer may include buffer 1218 in Figure 12. In some embodiments, after calculating the current WM keyframe, the current WM keyframe may be stored in a second WM buffer as a new preceding WM keyframe. Method 1300 may further include, after calculating the current WM keyframe, calculating the next WM keyframe around the autonomous vehicle based on additional map data (e.g., an updated lane graph), loading the next WM keyframe into the first WM buffer when it is calculated, converting the new preceding WM keyframe stored in the second WM buffer into a coordinate frame of the autonomous vehicle in a third time before the calculation of the next WM keyframe is completed, and converting the new preceding WM keyframe stored in the second WM buffer into a coordinate frame of the autonomous vehicle in a fourth time after the third time, before the calculation of the next WM keyframe is completed.

[0131] Figure 14 is a flowchart showing another method 1400 for generating a WM frame according to some embodiments of the present disclosure. Each block of method 1400 and / or other methods described herein includes a computation process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. The method may also be performed as computer-available instructions stored on a computer storage medium. The method may be provided, to name a few, by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or by plugging into another product. In some implementations, method 1400 and / or other methods described herein are stored in and / or performed by a vehicle computing system, for example, a vehicle computing system 120 as described elsewhere herein. For example, method 1400 may be performed at least in part by a WM generator 908 in a vehicle computing system 120 of an autonomous vehicle 150 or rig.

[0132] Method 1400 includes, in block 1402, calculating WM keyframes around an autonomous vehicle over time and at a first frequency based on map data. For example, block 1302 may include the WM producer 1202 of the WM generator 908 sequentially calculating the current WM keyframe from the current LaneGraph 1208 sequence at a first frequency.

[0133] Method 1400 includes generating WM frames in block 1404 by converting the most recent WM keyframe to an autonomous vehicle coordinate frame over time and at a second frequency greater than a first frequency, so that multiple WM frames are generated from each WM keyframe. For example, block 1404 may include the WM transformer 1206 of the WM generator 908 generating WM keyframes 1212 by converting sequential preceding (or most recent) WM keyframes 1214 to an autonomous vehicle coordinate frame over time and at a second frequency, so that multiple WM keyframes 1212 are generated from each preceding WM keyframe 1214.

[0134] In some implementations, method 1400 may further include determining a transformation to be applied to the most recent WM keyframe over time and at a second frequency. In this example, transforming the most recent (or preceding) WM keyframe to the autonomous vehicle's coordinate frame over time may include transforming the most recent WM keyframe to the autonomous vehicle's coordinate frame according to the determined transformation. For example, the WM attitude calculation node 1204 of the WM generator 908 may determine, for example, calculate, an effective transformation 1220 for the preceding WM keyframe 1214 over time based on the localization result 1214, and the preceding WM keyframe 1214 may be transformed to the autonomous vehicle's coordinate frame according to the effective transformation 1220.

[0135] Map data on which WM keyframes are calculated may include, for example, a lane graph such as one of the LaneGraphs described herein. Thus, calculating WM keyframes around an autonomous vehicle based on map data may include calculating WM keyframes around an autonomous vehicle based on a lane graph. Some embodiments of Method 1400 may further include generating a lane graph from a portion of an HD map, which includes incrementally increasing the collection of active lanes around an autonomous vehicle from a portion of an HD map, as described with respect to Figure 11A, for example. Some embodiments of Method 1400 may further include alternately feeding the currently incrementally increasing collection of active lanes into a first active lane buffer and a second active lane buffer. In this and other embodiments, the first active lane buffer may be populated with the currently incrementally increasing active lane collection when the most recent incrementally increasing collection of active lanes is stored in the second active lane buffer, and the second active lane buffer may be populated with the currently incrementally increasing active lane collection when the most recent incrementally increasing collection of active lanes is stored in the first active lane buffer.

[0136] Alternatively or in addition, Method 1400 may further include alternately loading WM keyframes into a first WM buffer and a second world model buffer. As an example, the first WM buffer may include buffer 1216 in Figure 12, and the second WM buffer may include buffer 1218 in Figure 12. The first WM buffer may be loaded with the WM keyframe currently being calculated when the most recent WM keyframe is stored in the second WM buffer. The second WM buffer may be loaded with the WM keyframe currently being calculated when the most recent WM keyframe is stored in the first WM buffer.

[0137] The compression performance in some of the examples described herein may be significant. It can be assumed that the size of each LaneGraph is significantly larger than the size of each transformation. In this example, the compression ratio R cThis can simply be the transmission interval of a LaneGraph with static geometry, or laneGraphIntervalFrames. The transmission interval laneGraphIntervalFrames may be related to the size of the RoadSegment roadSegmentLength and the vehicle speed vehicleSpeed. Specifically, the transmission interval laneGraphIntervalFrames can be determined according to equation 1:

number

[0138] Therefore, in either or both of methods 1300, 1400, or any other method described herein, the compression ratio of calculating WM keyframes at the first frequency relative to the second frequency may depend on the speed of the autonomous vehicle and may be at least 10, at least 40, at least 100, or any other value.

[0139] Exemplary autonomous vehicle Figure 15A shows an exemplary autonomous vehicle 1500 according to some embodiments of the present disclosure. The autonomous vehicle 1500 (or referred to herein as "vehicle 1500") may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first responder vehicles, shuttles, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments within the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). Mobile vehicle 1500 may have the capability to perform functions at one or more of the autonomous driving levels from Level 3 to Level 5. For example, depending on the embodiment, mobile vehicle 1500 may have the capability of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0140] The mobile vehicle 1500 may include components such as the chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. The mobile vehicle 1500 may include a propulsion system 1550, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 1550 may be connected to the drivetrain of the mobile vehicle 1500, which may include a transmission, in order to enable propulsion for the mobile vehicle 1500. The propulsion system 1550 may be controlled in response to receiving a signal from a throttle / accelerator 1552.

[0141] A steering system 1554, which may include a steering wheel, may be used to steer the vehicle 1500 (for example, along a desired course or route) when the propulsion system 1550 is operating (for example, when the vehicle is moving). The steering system 1554 may receive signals from the steering actuator 1556. The steering wheel may also be an option for fully automated (level 5) functionality.

[0142] The brake sensor system 1546 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1548 and / or the brake sensor.

[0143] The controller 1536, which may include one or more CPUs, a system-on-a-chip (SoC) 1504 (Figure 15C), and / or a GPU, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 1500. For example, the controller can send signals to operate the vehicle brakes via one or more brake actuators 1548, to operate the steering system 1554 via one or more steering actuators 1556, and / or to operate the propulsion system 1550 via one or more throttle / accelerators 1552. The controller 1536 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist the driver in driving the vehicle 1500. The controller 1536 may include a first controller 1536 for autonomous driving functions, a second controller 1536 for functional safety functions, a third controller 1536 for artificial intelligence functions (e.g., computer vision), a fourth controller 1536 for infotainment functions, a fifth controller 1536 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1536 may handle two or more of the aforementioned functions, and two or more controllers 1536 may handle a single function, and / or any combination thereof.

[0144] The controller 1536 can provide signals for controlling one or more components and / or systems of the mobile vehicle 1500 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example and without limitation, global navigation satellite system sensors 1558 (e.g., global positioning system sensors), RADAR sensors 1560, ultrasonic sensors 1562, LIDAR sensors 1564, inertial measurement unit (IMU) sensors 1566 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1596, stereo cameras 1568, wide-view cameras 1570 (e.g., fisheye cameras), infrared cameras 1572, surround cameras 1574 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1598, speed sensors 1544 (e.g., for measuring the speed of a moving vehicle 1500), vibration sensors 1542, steering sensors 1540, brake sensors 1546 (e.g., as part of a brake sensor system 1546), and / or other sensor types.

[0145] One or more of the controllers 1536 may receive inputs (represented, for example, by input data) from the instrument cluster 1532 of the mobile vehicle 1500 and provide outputs (represented, for example, by output data, display data, etc.) via a human-machine interface (HMI) display 1534, an audible annunciator, a loudspeaker, and / or other components of the mobile vehicle 1500. The outputs may include information such as mobile vehicle velocity, speed, time, map data (e.g., HD map 1522 in Figure 15C), location data (e.g., the location of the mobile vehicle 1500, such as on the map), direction, the locations of other mobile vehicles (e.g., occupied grids), and information about objects and the status of objects as grasped by the controller 1536. For example, the HMI display 1534 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or driving operations that the moving vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting exit 34B within 3.22 km (2 miles), etc.).

[0146] The mobile vehicle 1500 further includes a network interface 1524 that can communicate over one or more networks using one or more wireless antennas 1526 and / or a modem. For example, the network interface 1524 may have the capability to communicate over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The wireless antennas 1526 can also enable communication between objects in the environment (e.g., mobile vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWANs) such as LoRaWAN, SigFox.

[0147] Figure 15B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 1500 of Figure 15A according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different positions on the mobile vehicle 1500.

[0148] The camera type may include, but is not limited to, a digital camera that can be used with components and / or systems of the mobile vehicle 1500. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have the capability of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the capability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

[0149] In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0150] One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0151] A camera having a field of view that includes a portion of the environment in front of the moving vehicle 1500 (e.g., a forward-facing camera) may be used for surround view to help identify the forward path and obstacles and, with the help of one or more controllers 1536 and / or control SoCs, to help provide information essential for generating an occupied grid and / or determining a preferred moving vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as Lane Departure Warning (LDW), Autonomous Cruise Control (ACC), and / or traffic sign recognition.

[0152] Various cameras may be used in forward-facing configurations, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 1570, which can be used to capture objects entering the view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-view camera is shown in Figure 15B, any number of wide-view cameras 1570 may be present in the mobile vehicle 1500. In addition, long-range cameras 1598 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 1598 may also be used for object detection and classification, as well as basic object tracking.

[0153] One or more stereo cameras 1568 may also be included in a forward-facing configuration. The stereo camera 1568 may include an integrated control unit with an expandable processing unit that can provide programmable logic (e.g., FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the moving vehicle's environment, including distance estimates of all points in the image. An alternative stereo camera 1568 may include a compact stereo vision sensor that includes two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the moving vehicle to an object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 1568 may be used in addition to or instead of those described herein.

[0154] A camera having a field of view including a portion of the environment on the sides of the mobile vehicle 1500 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and generate side impact collision warnings. For example, surround cameras 1574 (e.g., four surround cameras 1574 as shown in Figure 15B) may be positioned around the mobile vehicle 1500. The surround cameras 1574 may include wide-view cameras 1570, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and to the sides of the mobile vehicle. In an alternative configuration, the mobile vehicle may use three surround cameras 1574 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround view camera.

[0155] A camera having a field of view that includes a portion of the environment behind the mobile vehicle 1500 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 1598, stereo camera 1568), infrared cameras 1572, etc., as described herein.

[0156] Figure 15C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 1500 of Figure 15A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0157] Each component, feature, and system of the mobile vehicle 1500 in Figure 15C is illustrated as being connected via a bus 1502. Bus 1502 may include a Controller Area Network (CAN) data interface (or, collectively, the "CAN bus"). CAN may also be a network within the mobile vehicle 1500 used to assist in the control of various features and functions of the mobile vehicle 1500, such as the operation of brakes, acceleration, steering, and windshield wipers. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other mobile vehicle status indicators. The CAN bus may be ASIL B compliant.

[0158] Bus 1502 is described herein as a CAN bus, but this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to, or as an alternative to, a CAN bus. In addition, a single line is used to represent bus 1502, but this is not intended to limit it. There may be any number of buses 1502, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1502 may be used to perform different functions and / or for redundancy. For example, a first bus 1502 may be used for collision avoidance and a second bus 1502 may be used for operation control. In any example, each bus 1502 may communicate with any of the components of the mobile vehicle 1500, and two or more buses 1502 may communicate with the same component. In some examples, each SoC 1504, each controller 1536, and / or each computer within the mobile vehicle may have access to the same input data (e.g., input from sensors in the mobile vehicle 1500) and may be connected to a common bus such as a CAN bus.

[0159] The mobile vehicle 1500 may include one or more controllers 1536, such as those described herein with respect to Figure 15A. The controllers 1536 may be used for a variety of functions. The controllers 1536 may be connected to any of the various other components and systems of the mobile vehicle 1500 and may be used for the control of the mobile vehicle 1500, the artificial intelligence of the mobile vehicle 1500, infotainment for the mobile vehicle 1500, and / or the like.

[0160] The mobile vehicle 1500 may include a system-on-a-chip (SoC) 1504. The SoC 1504 may include a CPU 1506, a GPU 1508, a processor 1510, a cache 1512, an accelerator 1514, a data store 1516, and / or other components and features not shown. The SoC 1504 can be used to control the mobile vehicle 1500 in various platforms and systems. For example, the SoC 1504 may be coupled in a system (e.g., a system of the mobile vehicle 1500) that has an HD map 1522 that can obtain map refreshes and / or updates from one or more servers (e.g., server 1578 in Figure 15D) via a network interface 1524.

[0161] The CPU 1506 may include a CPU cluster or CPU complex (also referred to as "CCPLEX"). The CPU 1506 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1506 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 1506 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU 1506 (e.g., CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of the CPU 1506 clusters to be active at any given time.

[0162] The CPU1506 can implement power management capabilities that include one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU1506 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. The processing core may support a simplified power state input sequence in software where the work is offloaded to the microcode.

[0163] The GPU1508 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU1508 may be programmable and efficient for parallel workloads. In some embodiments, the GPU1508 may be able to use an enhanced tensor instruction set. The GPU1508 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache with a storage capacity of 512KB). In some embodiments, the GPU1508 may include at least eight streaming microprocessors. The GPU1508 may be able to use a computer-based application programming interface (API). In addition, the GPU1508 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0164] The GPU1508 can be power-optimized for optimal performance in automotive and embedded use cases. For example, the GPU1508 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be a limitation, and the GPU1508 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. Not limited to, for example, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mixture of computation and addressing operations. A streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. A streaming microprocessor may also include a combined L1 data cache and shared memory unit to simplify programming while improving performance.

[0165] In some examples, the GPU1508 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used.

[0166] The GPU1508 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges across processors. In some examples, address translation service (ATS) support may be used to allow the GPU1508 to directly access the CPU1506 page table. In such examples, when the GPU1508 memory management unit (MMU) experiences a miss, an address translation request may be sent to the CPU1506. In response, the CPU1506 can examine its page table for virtual-to-real-address mapping and send the translation back to the GPU1508. As such, unified memory technology can enable a single, unified virtual address space for both the CPU1506 and GPU1508 memory, thereby simplifying GPU1508 programming and porting of applications to the GPU1508.

[0167] In addition, the GPU1508 may include an access counter that can record how often the GPU1508 accesses the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently.

[0168] The SoC1504 may include any number of caches 1512, including those described herein. For example, cache 1512 may include an L3 cache available to both the CPU 1506 and the GPU 1508 (e.g., connected to both the CPU 1506 and the GPU 1508). Cache 1512 may include a write-back cache that can record line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4MB or more, depending on the embodiment, although a smaller cache size may be used.

[0169] The SoC1504 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 1500 (for example, a processing DNN). In addition, the SoC1504 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU1506 and / or GPU1508.

[0170] The SoC1504 may include one or more accelerators 1514 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, the SoC1504 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the GPU1508 and to offload some of the tasks of the GPU1508 (e.g., to free up more cycles of the GPU1508 to perform other tasks). As an example, accelerator 1514 may be used for target workloads that are sufficiently stable to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). In this specification, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (for example, as used for object detection).

[0171] Accelerator 1514 (e.g., hardware acceleration cluster) may include a deep learning accelerator (DLA). A DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the DLA can provide more performance per millisecond than a general-purpose GPU and significantly exceed the performance of a CPU. The TPU can perform several functions, including, for example, single-instance convolutional functions supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0172] DLA can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and mobile vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0173] DLA can perform any function of GPU1508, and by using inference accelerators, for example, a designer can target either DLA or GPU1508 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU1508 and / or other accelerators 1514.

[0174] The accelerator 1514 (for example, a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0175] A RISC core can interact with an image sensor (for example, the image sensor of one of the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may contain any amount of memory. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0176] DMA can enable PVA components to access system memory independent of the CPU 1506. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0177] A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed.

[0178] Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, two or more vector processors included in a single PVA can run the same computer vision algorithm, but on different areas of an image. In other examples, the vector processors included in a particular PVA can run different computer vision algorithms simultaneously on the same image, or even run different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety.

[0179] The accelerator 1514 (for example, a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 1514. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, for example, and not limited to. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (for example, using an APB).

[0180] A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0181] In some embodiments, the SoC1504 may include a real-time ray tracing hardware accelerator, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0182] Accelerators 1514 (e.g., hardware accelerator clusters) have diverse applications for autonomous driving. PVAs may also be programmable vision accelerators that can be used in critical processing stages in ADAS and autonomous vehicles. The capabilities of PVAs are suitable for areas of algorithms requiring predictable processing at low power and low latency. In other words, PVAs perform well in semi-high density or high density typical computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, because PVAs are efficient in object detection and integer computation, in relation to platforms for autonomous vehicles, PVAs are designed to run classic computer vision algorithms.

[0183] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. While semi-global matching-based algorithms may be used in some examples, this is not intended to be a limitation. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras.

[0184] In some applications, PVA can be used to perform high-density optical flow. For example, PVA can be used to process raw RADAR data (e.g., using a 4D fast Fourier transform) to provide a processed RADAR signal before emitting the next RADAR pulse. In other applications, PVA can be used for flight depth processing, for example, by processing the raw time of flight data to provide the processed time of flight data.

[0185] DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing the relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the moving vehicle to automatically apply the emergency brakes, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for the AEB. DLA can run a neural network that devolves the confidence values. The neural network can accept at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimation (e.g., from another subsystem), object orientation, distance, inertial measurement unit (IMU) sensor output (e.g., IMU) output (e.g., IMU) which correlates with 3D position estimation of the moving object (e.g., IMU) sensor 1566, and others.

[0186] The SoC1504 may include a data store 1516 (for example, memory). The data store 1516 may also be the on-chip memory of the SoC1504 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 1516 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 1516 may include an L2 or L3 cache 1512. References to the data store 1516 may include references to memory associated with the PVA, DLA, and / or other accelerators 1514, as described herein.

[0187] The SoC1504 may include one or more processors 1510 (e.g., embedded processors). The processors 1510 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC1504 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, assistance with system low-power state transitions, management of SoC1504 thermal and temperature sensors, and / or management of SoC1504 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC1504 may use the ring oscillators to detect the temperatures of the CPU 1506, GPU 1508, and / or accelerator 1514. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine and place the SoC1504 into a lower power state and / or put the mobile vehicle 1500 into chauffeur safe shutdown mode (for example, safely shut down the mobile vehicle 1500).

[0188] The processor 1510 may further include a set of integrated processors that can perform the functions of an audio processing engine. The audio processing engine may also be an audio subsystem enabling full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0189] The processor 1510 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0190] The processor 1510 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic for detecting any differences between their operations.

[0191] The processor 1510 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0192] The processor 1510 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0193] The processor 1510 may include a video image synthesizer, which may also be a processing block (for example, implemented on a microprocessor) that implements post-video processing functions required by the video playback application to produce the final image for the player window. The video image synthesizer can perform lens distortion correction on the wide-view camera 1570, the surround camera 1574, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and appropriately respond to in-cabin events. The in-cabin system can activate cellular services and perform lip-reading to make phone calls, transcribe emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are otherwise disabled.

[0194] A video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0195] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, so that the GPU1508 is not required to continuously render new surfaces. Even when the GPU1508 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU1508 to improve performance and responsiveness.

[0196] The SoC1504 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC1504 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0197] The SoC1504 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC1504 may be used to process data from cameras (connected, for example, via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., LiDAR sensor 1564, RADAR sensor 1560, etc., which may be connected via Ethernet®), data from bus 1502 (e.g., speed of vehicle 1500, steering wheel position, etc.), and data from GNSS sensor 1558 (connected, for example, via Ethernet® or CAN bus). The SoC1504 may further include a dedicated high-performance mass storage controller which may include its own DMA engine and which may be used to free up CPU 1506 from routine data management tasks.

[0198] The SoC1504 may also be an inter-terminal platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The SoC1504 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 1514 is coupled with the CPU 1506, the GPU 1508, and the data store 1516 can provide a fast and efficient platform for autonomous vehicles at levels 3-5.

[0199] Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0200] In contrast to conventional systems, by providing CPU complexes, GPU complexes, and hardware acceleration clusters, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or a CNN running on a dGPU (e.g., GPU1520) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of signs and passing that semantic understanding to a route planning module running on the CPU complex.

[0201] As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network, informing the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU1508, for example.

[0202] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of the vehicle 1500. An always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and, in security mode, to stop the vehicle when the owner leaves the vehicle. In this way, the SoC 1504 provides security against theft and / or vehicle hijacking.

[0203] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from microphone 1596. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 1504 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In a preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the moving vehicle is operating, as identified by GNSS sensor 1558. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, and when in the United States, the CNN would attempt to identify only North American sirens. After an emergency vehicle is detected, the control program may be used, with the assistance of the ultrasonic sensor 1562, to perform an emergency vehicle safety routine, such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle, until the emergency vehicle has passed.

[0204] The vehicle may include a CPU 1518 (e.g., a separate CPU, or dCPU) which can be connected to the SoC 1504 via a high-speed interconnect (e.g., PCIe). The CPU 1518 may include, for example, an x86 processor. The CPU 1518 may be used to perform any of a variety of functions, including, for example, mediating the consequences of a potential mismatch between ADAS sensors and the SoC 1504, and / or monitoring the status and condition of the controller 1536 and / or the infotainment SoC 1530.

[0205] The mobile vehicle 1500 may include a GPU 1520 (e.g., a separate GPU, or dGPU) which can be connected to the SoC 1504 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 1520 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from the mobile vehicle 1500's sensors (e.g., sensor data).

[0206] Mobile vehicle 1500 may further include a network interface 1524 which may include one or more wireless antennas 1526 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna and a Bluetooth antenna). The network interface 1524 may be used to enable wireless connectivity to a cloud over the Internet (e.g., a server 1578 and / or other network devices), to other mobile vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other mobile vehicles, a direct link may be established between two mobile vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). The direct link may be provided using a mobile-to-mobile communication link. The mobile-to-mobile communication link may provide mobile vehicle 1500 information about mobile vehicles in close proximity to mobile vehicle 1500 (e.g., mobile vehicles in front of, beside, and / or behind mobile vehicle 1500). This function may also be part of the joint adaptive cruise control function of the mobile vehicle 1500.

[0207] The network interface 1524 may include an SoC that provides modulation and demodulation functions and enables the controller 1536 to communicate over a wireless network. The network interface 1524 may include a radio frequency front end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. Frequency conversion can be performed through well-known processes and / or using a superheterodyne process. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0208] The mobile unit 1500 may further include a data store 1528 which may include storage outside the chip (for example, outside the SoC 1504). The data store 1528 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0209] The vehicle 1500 may further include GNSS sensors 1558 (e.g., GPS and / or supported GPS sensors) to assist mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 1558 may be used, including, but not limited to, GPS using a USB connector with Ethernet® to a serial (RS-232) bridge.

[0210] The mobile vehicle 1500 may further include a RADAR sensor 1560. The RADAR sensor 1560 may be used by the mobile vehicle 1500 for long-range mobile vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 1560 may use CAN and / or bus 1502 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 1560) using Ethernet® access for accessing raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 1560 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor may be used.

[0211] The RADAR sensor 1560 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 1560 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include monostatic multimodal RADARs having multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record around the moving vehicle 1500 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of moving vehicles entering or leaving the lane of the moving vehicle 1500.

[0212] As an example, a medium-range RADAR system may include a range of up to 1560 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1550 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the moving vehicle.

[0213] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0214] The mobile vehicle 1500 may further include ultrasonic sensors 1562. Positioned on the front, rear, and / or sides of the mobile vehicle 1500, the ultrasonic sensors 1562 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 1562 may be used, and different ultrasonic sensors 1562 may be used for detection at different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 1562 may operate at a functional safety level of ASIL B.

[0215] The mobile vehicle 1500 may include a LiDAR sensor 1564. The LiDAR sensor 1564 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 1564 may also have a functional safety level of ASIL B. In some examples, the mobile vehicle 1500 may include multiple LiDAR sensors 1564 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch).

[0216] In some examples, the LIDAR sensor 1564 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 1564 may have an advertised range of approximately 100m, for example, with an accuracy of 2cm to 3cm and support for 100Mbps Ethernet® connectivity. In some examples, one or more non-protruding LIDAR sensors 1564 may be used. In such examples, the LIDAR sensor 1564 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a mobile vehicle 1500. In such examples, the LIDAR sensor 1564 may have a range of 200m even for low-reflection objects and can provide a field of view up to 120 degrees horizontal and 35 degrees vertical. A front-mounted LIDAR sensor 1564 may be configured for a horizontal field of view between 45 and 135 degrees.

[0217] In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around a moving vehicle up to approximately 200m. The flash LiDAR unit includes receptors that record the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the moving vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the moving vehicle. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1564 may be less susceptible to motion blur, vibration, and / or shock.

[0218] The mobile vehicle may further include an IMU sensor 1566. In some examples, the IMU sensor 1566 may be positioned in the center of the rear axle of the mobile vehicle 1500. The IMU sensor 1566 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 1566 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 1566 may include an accelerometer, a gyroscope, and a magnetometer.

[0219] In some embodiments, the IMU sensor 1566 may be implemented as a miniature, high-performance GPS-aided inertial navigation system (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 1566 may enable the moving vehicle 1500 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS to the IMU sensor 1566. In some embodiments, the IMU sensor 1566 and the GNSS sensor 1558 may be combined in a single integrated unit.

[0220] The mobile vehicle may include a microphone 1596 placed inside and / or around the mobile vehicle 1500. The microphone 1596 may, among other things, be used for emergency vehicle detection and identification.

[0221] The mobile vehicle may further include any number of camera types, including a stereo camera 1568, a wide-view camera 1570, an infrared camera 1572, a surround camera 1574, a long-range and / or medium-range camera 1598, and / or other camera types. The cameras may be used to capture image data around the entire exterior surface of the mobile vehicle 1500. The type of camera used will depend on the embodiment and requirements of the mobile vehicle 1500, and any combination of camera types may be used to achieve the required coverage around the mobile vehicle 1500. In addition, the number of cameras may vary depending on the embodiment. For example, the mobile vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 15A and 15B.

[0222] The mobile vehicle 1500 may further include a vibration sensor 1542. The vibration sensor 1542 can measure vibrations of components of the mobile vehicle, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 1542 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0223] The mobile vehicle 1500 may include an ADAS system 1538. In some examples, the ADAS system 1538 may include a System of Control (SoC). The ADAS system 1538 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0224] The ACC system may utilize a radar sensor 1560, a lithium-ion sensor 1564, and / or a camera. The ACC system may include longitudinal ACC and / or transverse ACC. Longitudinal ACC monitors and controls the distance of vehicle 1500 to the vehicle immediately in front of it and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Transverse ACC performs distance maintenance and advises vehicle 1500 to change lanes when necessary. Transverse ACC is related to other ADAS applications such as LC and CWS.

[0225] CACC uses information from other vehicles that can be received from other vehicles via a wireless link through the network interface 1524 and / or wireless antenna 1526, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., the vehicle immediately in front of vehicle 1500, in the same lane as vehicle 1500), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about the vehicle ahead of vehicle 1500, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

[0226] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 1560, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual warnings, vibrations, and / or quick brake pulses.

[0227] An AEB system can detect an imminent forward collision with another moving vehicle or other object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or impending collision braking.

[0228] The LDW system warns the driver when the vehicle 1500 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0229] The LKA system is a modified version of the LDW system. The LKA system provides steering input or braking to correct the vehicle 1500 when it begins to drift out of its lane.

[0230] The BSW system detects and warns the driver of a moving vehicle in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system may use a rear-facing camera and / or radar sensor 1560, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback components such as a display, speaker, and / or vibration component.

[0231] The RCTW system can provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rear-view camera while the vehicle 1500 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 1560, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0232] Conventional ADAS systems warn the driver and allow the driver to determine whether a safe condition truly exists and act accordingly. However, conventional ADAS systems have sometimes tended to produce misjudgments that, while not usually catastrophic, can be troubling and distracting to the driver. In the autonomous vehicle 1500, however, if the results are contradictory, the vehicle 1500 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 1536 or the second controller 1536). For example, in some embodiments, the ADAS system 1538 may also be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. The backup computer rationality monitor can run a variety of redundant software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 1538 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation.

[0233] In some implementations, a primary computer may be configured to provide a supervising MCU with a reliability score indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0234] The supervisor MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when the secondary computer is providing a false alarm. Thus, the neural network in the supervisor MCU can learn when the output of the secondary computer is reliable and when it is unreliable. For example, when the secondary computer is a radar-based FCW system, the neural network in the supervisor MCU can learn when the FCW is identifying metal objects that are not actually dangerous, such as sewer grates or manhole covers that trigger an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisor MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments involving a neural network running on the supervisor MCU, the supervisor MCU may include at least one of a DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC1504.

[0235] In other examples, ADAS system 1538 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware used by the primary computer is not causing a critical error.

[0236] In some examples, the output of the ADAS system 1538 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 1538 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment.

[0237] The mobile vehicle 1500 may further include an infotainment SoC 1530 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more separate components. The infotainment SoC 1530 may include a combination of hardware and software that can be used to provide the mobile vehicle 1500 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, fuel level, total mileage, brake fuel level, oil level, door open / close, air filter information, and other mobile vehicle-related information). For example, the infotainment SoC 1530 may include wireless, disc player, navigation system, video player, USB and Bluetooth connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 1534, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1530 may be further used to provide information (for example, visual and / or audible) to the user of the vehicle, such as information from the ADAS system 1538, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0238] The infotainment SoC 1530 may include GPU functionality. The infotainment SoC 1530 can communicate with other devices, systems, and / or components of the vehicle 1500 via bus 1502 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 1530 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 1536 (e.g., the primary and / or backup computer of the vehicle 1500). In such examples, the infotainment SoC 1530 can put the vehicle 1500 into a chauffeur-safe stop mode as described herein.

[0239] The mobile vehicle 1500 may further include an instrument cluster 1532 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1532 may include a controller and / or a supercomputer (e.g., a separate controller or supercomputer). The instrument cluster 1532 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some examples, information may be displayed and / or shared between the infotainment SoC 1530 and the instrument cluster 1532. In other words, the instrument cluster 1532 may be included as part of the infotainment SoC 1530, and vice versa.

[0240] Figure 15D is a system diagram of communication between the cloud-based server in Figure 15A and an exemplary autonomous vehicle 1500, according to some embodiments of the present disclosure. System 1576 may include a server 1578, a network 1590, and a mobile vehicle including the mobile vehicle 1500. Server 1578 may include two or more GPUs 1584(A) to 1584(H) (collectively referred to herein as GPU 1584), PCIe switches 1582(A) to 1582(H) (collectively referred to herein as PCIe switch 1582), and / or CPUs 1580(A) to 1580(B) (collectively referred to herein as CPU 1580). The GPUs 1584, CPUs 1580, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 1588 and / or PCIe connections 1586 developed by NVIDIA. In some examples, the GPU1584 is connected via NVLink and / or NVSwitch SoCs, and the GPU1584 and PCIe switch 1582 are connected via PCIe interconnects. Eight GPU1584s, two CPU1580s, and four PCIe switch 1582s are illustrated, but this is not intended to be an limitation. Depending on the embodiment, each server 1578 may contain any number of GPU1584s, CPU1580s, and / or PCIe switches. For example, server 1578 may contain eight, sixteen, thirty-two, and / or more GPU1584s, respectively.

[0241] Server 1578 can receive image data from mobile vehicles via network 1590, representing images showing unexpected or altered road conditions, such as recently commenced road construction. Server 1578 can transmit map information 1594, including information about traffic and road conditions, to mobile vehicles via network 1590, including information about neural networks 1592, updated neural networks 1592, and / or map information 1594. Updates to map information 1594 may include updates to HD map 1522, including information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural networks 1592, updated neural networks 1592, and / or map information 1594 may have arisen from new training and / or experience represented in data received from any number of mobile vehicles in the environment, and / or based on training performed in a data center (e.g., using server 1578 and / or other servers).

[0242] Server 1578 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a mobile device and / or in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, it may be used by the mobile vehicle (for example, transmitted to the mobile vehicle via network 1590), and / or the machine learning model may be used by server 1578 to remotely monitor the mobile vehicle.

[0243] In some examples, Server 1578 can receive data from a mobile vehicle and apply it to a state-of-the-art real-time neural network for real-time intelligent inference. Server 1578 may include deep learning supercomputers and / or dedicated AI computers powered by GPU 1584, such as the DGX and DGX Station Machines developed by NVIDIA. However, in some examples, Server 1578 may include deep learning infrastructure that uses only CPU-powered data centers.

[0244] The deep learning infrastructure of server 1578 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware in mobile vehicle 1500. For example, the deep learning infrastructure can receive periodic updates from mobile vehicle 1500, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by mobile vehicle 1500, and if the results do not match and the infrastructure concludes that the AI ​​in mobile vehicle 1500 is not functioning properly, server 1578 can send a signal to mobile vehicle 1500 instructing the mobile vehicle's fail-safe computer to infer control, notify passengers, and complete a safe parking operation.

[0245] For inference, server 1578 may include GPU 1584 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when high performance is not required, a server powered by a CPU, FPGA, and other processors may be used for inference.

[0246] Exemplary computing devices Figure 16 is a block diagram of an example of a computing device 1600 suitable for use in implementing some embodiments of the present disclosure. The computing device 1600 may include an interconnection system 1602 that indirectly or directly connects the following devices: memory 1604, one or more central processing units (CPUs) 1606, one or more graphics processing units (GPUs) 1608, a communication interface 1610, I / O ports 1612, input / output components 1614, a power supply unit 1616, one or more presentation components 1618 (e.g., a display), and one or more logic units 1620.

[0247] The various blocks in Figure 16 are shown connected by lines via the interconnection system 1602, but this is not intended to be restrictive and is simply for clarity. For example, in some embodiments, a presentation component 1618, such as a display device, could be considered an I / O component 1614 (for example, if the display is a touchscreen). As another example, the CPU 1606 and / or GPU 1608 may include memory (for example, memory 1604 may represent a storage device in addition to the memory of the GPU 1608, CPU 1606, and / or other components). In other words, the computing devices in Figure 16 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” “augmented reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 16 and are therefore not distinguished.

[0248] The interconnection system 1602 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1602 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 1606 may be directly connected to the memory 1604. Furthermore, the CPU 1606 may be directly connected to the GPU 1608. Where direct or point-to-point connections exist between components, the interconnection system 1602 may include PCIe links for implementing the connections. In these examples, the PCI bus does not need to be included in the computing device 1600.

[0249] Memory 1604 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 1600. The computer-readable media may include both volatile and non-volatile media, and removable and non-removable media. For example, but not limited to, the computer-readable media may include computer storage media and communication media.

[0250] Computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1604 may store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 1600. In this specification, computer storage media does not include signals themselves.

[0251] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” may refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. For example, but not limited to, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0252] The CPU 1606 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1600 to execute one or more of the methods and / or processes described herein. The CPU 1606 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads concurrently. The CPU 1606 may include any type of processor, and depending on the type of computing device 1600 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 1600, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1600 may include one or more CPUs 1606 in one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor.

[0253] In addition to or instead of the CPU 1606, the GPU 1608 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1600 to execute one or more of the methods and / or processes described herein. One or more of the GPUs 1608 may be an integrated GPU (for example, with one or more of the CPUs 1606, and / or one or more of the GPUs 1608 may be discrete GPUs). In embodiments, one or more of the GPUs 1608 may be coprocessors of one or more of the CPUs 1606. The GPUs 1608 may be used by the computing device 1600 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPUs 1608 may be used for GPU-based general-purpose computing (GPGPU). It can be used for a GPU. The GPU1608 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU1608 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from CPU1606 received via the host interface). The GPU1608 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 1604. The GPU1608 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or via a switch (for example, using NVSwitch). When coupled together, each GPU1608 can generate pixel data or GPGPU data for different parts of an output or different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs.

[0254] In addition to or instead of the CPU 1606 and / or GPU 1608, the logical unit 1620 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 1600 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 1606, GPU 1608, and / or the logical unit 1620 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logical units 1620 may be part of and / or integrated with one or more of the CPU 1606 and / or GPU 1608, and / or one or more of the logical units 1620 may be discrete components of the CPU 1606 and / or GPU 1608 or otherwise external to them. In embodiments, one or more of the logical units 1620 may be coprocessors of one or more of the CPU 1606 and / or one or more of the GPU 1608.

[0255] Examples of the logic unit 1620 include one or more processing cores and / or components thereof, such as Tensor Cores (TCs), Tensor Processing Units (TPUs), Pixel Visual Cores (PVCs), Vision Processing Units (VPUs), Graphics Processing Clusters (GPCs), Texture Processing Clusters (TPCs), Streaming Multiprocessors (SMs), Tree Traversal Units (TTUs), Artificial Intelligence Accelerators (AIAs), Deep Learning Accelerators (DLAs), Logical Units (ALUs), Application-Specific Integrated Circuits (ASICs), Floating-Point Units (FPUs), I / O elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) elements, and / or similar.

[0256] The communication interface 1610 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1600 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 1610 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0257] I / O port 1612 can enable the computing device 1600 to be logically connected to other devices, including I / O components 1614, presentation components 1618, and / or other components, some of which can be built into (e.g., integrated into) the computing device 1600. Exemplary I / O components 1614 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 1614 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of voice recognition, stylus recognition, face recognition, biometric recognition, on-screen and beside-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 1600 (as described in more detail below). The computing device 1600 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 1600 may include accelerometers or gyroscopes to enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 1600 to render immersive augmented reality or virtual reality.

[0258] The power supply device 1616 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 1616 can provide power to the computing device 1600 to enable the components of the computing device 1600 to operate.

[0259] The presentation component 1618 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 1618 can receive data from other components (e.g., GPU 1608, CPU 1606, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0260] This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0261] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0262] Furthermore, the use of the term "based on" should not be interpreted as "based solely on" or "based only on." Rather, the first element "based on" the second element includes instances where the first element is based solely on the second element or on the second element and one or more additional elements.

[0263] The subject matter of this disclosure is described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the claimed subject matter may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, expressed.

Claims

1. A step of calculating the current keyframe, wherein the current keyframe represents the area around the autonomous vehicle at the current time based on map data; In a first time step prior to completing the calculation of the current keyframe, a step of generating a first world model frame compressed based on the preceding keyframe by compressing it based on the preceding keyframe and converting it into the coordinate frame of the autonomous vehicle, After the first time, and before the calculation of the current keyframe is completed, in a second time, the steps of compressing the preceding keyframe and converting it into the coordinate frame of the autonomous vehicle to generate a second world model frame compressed based on the preceding keyframe, and Methods that include...

2. The method according to claim 1, further comprising the step of determining a transformation to be applied to the preceding keyframe, wherein the step of transforming the preceding keyframe into the coordinate frame of the autonomous vehicle includes the step of transforming the preceding keyframe into the coordinate frame of the autonomous vehicle using the determined transformation.

3. The method according to claim 1, wherein the step of calculating the current keyframe includes the step of calculating the current keyframe based on a lane graph representing one or more lanes in the area around the autonomous vehicle at the current time.

4. The method according to claim 3, further comprising the step of generating the lane graph from a portion of a high-resolution (HD) map, wherein the generating step includes incrementally increasing the collection of effective lanes around the autonomous vehicle using the portion of the HD map.

5. The step of incrementally increasing the collection of effective lanes around the autonomous vehicle is performed for at least two sequential frames within a plurality of frames, A step of retaining in the collection at least one lane from a preceding frame remaining within the horizon of the autonomous vehicle, wherein the horizon is represented as the outer perimeter of a predetermined area centered on the autonomous vehicle; The steps include adding each lane within the horizon to the collection, The steps of removing from the collection at least one lane that is no longer within the horizon but is held or added to the preceding frame. The method according to claim 4, including the method described in claim 4.

6. The method according to claim 4, wherein the incrementally increased collection of the active lanes is stored using a first active lane buffer, and the method further includes the step of loading the current incrementally increased collection of the active lanes into a second active lane buffer.

7. The method according to claim 1, wherein the preceding keyframe is stored using a first world model buffer, and the method further includes the step of loading the current keyframe into a second world model buffer.

8. After the step of calculating the current keyframe, the current keyframe is stored in the second world model buffer as a new preceding keyframe, and the method further after the step of calculating the current keyframe, The steps include: calculating the next keyframe representing the area around the autonomous vehicle based at least partially on additional map data; The steps include inserting the next keyframe into the first world model buffer, In a third time period prior to the completion of the calculation of the next keyframe, the steps include: converting the new preceding keyframe stored using the second world model buffer to the coordinate frame of the autonomous vehicle; After the third time, and before the calculation of the next keyframe is completed, the fourth time involves the steps of converting the new preceding keyframe stored in the second world model buffer to the coordinate frame of the autonomous vehicle. The method according to claim 7, including the method described in claim 7.

Images