INTELLIGENT MANAGEMENT AND CLEANING OF MAP DATA FOR AUTONOMOUS SYSTEMS AND APPLICATIONS

Abstract

In various examples, a technique for managing map data stored in a location-aware system includes determining a corresponding set of attributes for each of a plurality of map data units stored in a working memory within the location-aware system. For each map data unit included in the plurality of map data units, the technique calculates a priority number value for the map data unit based on the set of attributes corresponding to the map data unit. The technique also includes determining, based on the plurality of priority number values for the plurality of map data units, one or more map data units to be evicted from the working memory.The technique further includes causing at least a portion of the one or more map data units to be deleted from memory in response to receiving one or more new map data units for storage in memory.

Description

BACKGROUND

[0001] Autonomous vehicles, semi-autonomous vehicles, and / or other types of modern vehicles are typically equipped with navigation systems that use geographically indexed map data to provide users with geolocation, navigation, planning, and / or other types of location-based services. This map data may be organized into layers representing different data types, such as (but not limited to) imagery, roads, highways, road markings, traffic signs, radar points, traffic conditions, weather conditions, landmarks, topography, and / or metadata. This map data may additionally be subdivided along spatial boundaries into discrete units, such as map tiles, which represent contiguous geographic regions within a map. These discrete units may further be defined at multiple "zoom" levels, representing different levels of detail.For example, a map tile assigned to a low zoom level may represent a relatively large area such as a city, while a map tile assigned to a high zoom level may represent a single building or a city block.

[0002] During vehicle operation, discrete units of map data are typically downloaded by the vehicle from a server and stored in non-volatile flash memory on a system-on-chip (SoC). As the vehicle travels to new geographic areas, additional map data units (e.g., tiles) for the new geographic areas are downloaded and stored in the SoC so that the vehicle can continue to provide navigation services. Because the memory on the SoC has a limited storage capacity, at least some of the existing map data stored on the SoC generally needs to be evicted to make room for the just-downloaded map data.

[0003] Conventional approaches to updating map data used in vehicle navigation typically involve replacing the first available unused map tiles on disk and / or in a directory tree with new map data. These conventional approaches may also, or instead, use a least-recently-used (LRU) technique to replace map data that has not been used for the longest time with recently downloaded map data. However, the naive and rudimentary nature of these conventional approaches may cause existing map data likely to be needed in the near future to be deleted, creating additional resource overhead and potential latency when the deleted map data is subsequently re-downloaded.

[0004] Therefore, there is a need for more effective techniques to improve the cache and memory management of map data used in vehicle navigation. SUMMARY

[0005] The invention is defined by the claims. To illustrate the invention, aspects and embodiments are described herein, which may or may not be within the scope of the claims.

[0006] Embodiments of the present disclosure relate to the intelligent management and clearing of map data. The techniques described herein include determining a corresponding set of attributes for each of a plurality of map data units stored in memory within the location-aware system. The techniques also include, for each map data unit included in the plurality of map data units, calculating a priority numeric value for the map data unit based on the set of attributes corresponding to the map data unit.The technique also includes determining, based on the plurality of priority number values for the plurality of map data units, one or more map data units to be evicted from memory, and causing at least a portion of the one or more map data units to be deleted from memory in response to receiving one or more new map data units for storage in memory.

[0007] A technical advantage of the disclosed techniques relative to prior approaches is the ability to effectively utilize limited memory for storing and updating map data in a location-aware system. In this regard, the techniques prioritize map data units for eviction based on various attributes relevant to the subsequent use of the map data units and / or the cost associated with downloading the map data units, thereby reducing the likelihood that evicted map data must be subsequently re-downloaded. Consequently, the disclosed techniques improve data reuse, network utilization, and resource overhead compared to conventional, naive approaches for managing and updating stored map data.

[0008] The invention is defined by the claims. To illustrate the invention, aspects and embodiments are described herein, which may or may not be within the scope of the claims.

[0009] The disclosure extends to all novel aspects or features described and / or illustrated herein.

[0010] Further features of the disclosure are characterized by the independent and dependent claims.

[0011] Any feature in one aspect of the disclosure may be applied to other aspects of the disclosure in any suitable combination. In particular, method aspects may be applied to device or system aspects, and vice versa.

[0012] Furthermore, features implemented in hardware may also be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly.

[0013] Any system or device feature described herein may also be provided as a method feature, and vice versa. Functionally described system and / or device aspects (including means and function features) may alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0014] It should also be understood that certain combinations of the various features described and defined in any aspects of the disclosure may be implemented and / or provided and / or used independently of one another.

[0015] The disclosure also provides computer programs and computer program products comprising software code configured to, when executed on a data processing device, perform any of the methods described herein and / or embody any of the device and system features described herein, including any or all component steps of a method.

[0016] The disclosure also provides a computer or computing system (including networked or distributed systems) having an operating system that supports a computer program for performing the methods described herein and / or for embodying the device or system features described herein.

[0017] The disclosure also provides a computer-readable medium having stored thereon one or more of the aforementioned computer programs.

[0018] The disclosure also provides a signal carrying one or more of the aforementioned computer programs.

[0019] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0020] Aspects and embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present systems and methods for intelligent map data management and clearing are described in detail below with reference to the attached drawings. They show: Fig. 1 illustrates a computing device configured to implement one or more aspects of various embodiments; Fig. 2 a more detailed illustration of the management machine and the processing machine of Fig. 1 according to various embodiments; Fig. 3A the exemplary operation of the management machine of Fig. 1 in determining a set of eviction candidates according to various embodiments; Fig. 3B the exemplary operation of the management machine of Fig. 1 in determining a set of eviction candidates according to various embodiments; Fig. 4A is a flowchart of a method for managing map data stored in a location-aware system, according to various embodiments; Fig. 4B is a flowchart of a method for generating priority number values for map data units using a machine learning model according to various embodiments; Fig. 5A is an illustration of an exemplary autonomous vehicle, according to some embodiments of the present disclosure; Fig. 5B shows an example of camera locations and fields of view for the exemplary autonomous vehicle from Fig. 5A, according to some embodiments of the present disclosure; Fig. 5C is a block diagram of an exemplary system architecture for the exemplary autonomous vehicle of Fig. 5A, according to some embodiments of the present disclosure; Fig. 5D shows a system diagram for communication between one or more cloud-based servers and the exemplary autonomous vehicle Fig. 5A, according to some embodiments of the present disclosure; Fig. 6 is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure; and Fig. 7 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Systems and methods relating to the intelligent management and clearing of map data for autonomous and semi-autonomous systems and applications are disclosed. Although the present disclosure is described with respect to an exemplary autonomous or semi-autonomous vehicle or machine 500 (alternatively referred to herein as "vehicle 500," "ego-vehicle 500," "machine 500," or "ego-machine 500"), an example of which is provided with respect to Fig. 5A-5D), this is not limiting. For example, the systems and methods described herein may be used without limitation by location-aware systems capable of detecting, calculating, and / or utilizing the geographic position of a person, a mobile device, and / or a moving object. These location-aware systems may be implemented in non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g.,be included in and / or used in conjunction with one or more adaptive driver assistance systems (ADAS), autonomous vehicles or machines, guided and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, hydrofoils, boats, shuttle vehicles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Furthermore, although the present disclosure may be described with respect to the management and clearing of map data for autonomous or semi-autonomous machine applications, this is not to be construed as limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology areas in which data management and / or clearing may be used.

[0023] As explained herein, map data used by a vehicle to provide location-based services is typically downloaded and stored in limited on-chip memory. When the vehicle travels to a new geographic area, additional map data for the new geographic area is downloaded and stored in memory so that the vehicle can continue to provide these services in the new geographic areas. However, the limited storage capacity of on-chip memory generally requires that at least some existing map data in memory be evicted to make room for the map data just downloaded. At the same time, traditional, non-intelligent approaches to evicting stored map data in response to just downloaded map data may cause existing map data likely located in (e.g.,needed in the near future, resulting in additional resource overhead and potential latency when the deleted map data is subsequently re-downloaded.

[0024] To improve the management and use of map data in location-aware systems with limited memory, the disclosed techniques calculate priority number values for discrete units of map data downloaded in response to vehicle movements and stored in the memory of an in-vehicle SoC (or other location-aware system, hardware type, memory type, etc.). These map data units may include map tiles, layers within map tiles, and / or other discrete pieces of map data. The priority number values may be calculated based on attributes of the map data units, such as (but not limited to) recency of use, frequency of use, distance from a current location of the location-aware system, overlap with a frequent or known route of the location-aware system, a map data unit size, a time of day, and / or a day of the week.Each priority number value may represent a probability of use for a corresponding map data unit and / or another measure of a potential "cost" associated with clearing the map data unit. One or more map data units with priority number values representing the lowest potential cost or cost for clearing may then be deleted in response to new map data being stored in memory.

[0025] A technical advantage of the disclosed techniques relative to prior approaches is the ability to effectively utilize limited memory for storing and updating map data in a location-aware system. In this regard, the techniques prioritize map data units for eviction based on various attributes relevant to the subsequent use of the map data units and / or the cost associated with downloading the map data units, thereby reducing the likelihood that evicted map data must be subsequently re-downloaded. Consequently, the disclosed techniques improve data reuse, network utilization, and resource overhead compared to conventional, naive approaches for managing and updating stored map data.

[0026] Fig. 1 illustrates a computing device 100 configured to implement one or more aspects of various embodiments. In at least one embodiment, the computing device 100 includes a desktop computer, a laptop computer, a smartphone, a personal digital assistant (PDA), a tablet computer, a server, one or more virtual machines, an embedded system, one or more systems on one or more chips, a vehicle-mounted computing device, and / or any other type of computing device configured to receive input, process data, and optionally display images, and suitable for implementing one or more embodiments. The computing device 100 is configured to run a management engine 122 and a processing engine 124, which may be located in a memory 116.It should be noted that the computing device described herein is illustrative, and all other technically feasible configurations are within the scope of the present disclosure. For example, multiple instances of management engine 122 and / or processing engine 124 may be executed on a set of nodes in a distributed and / or cloud computing system to implement the functionality of computing device 100.

[0027] In one embodiment, computing device 100 includes, without limitation, a bus 112 connecting one or more processors 102, an input / output (I / O) device interface 104 coupled to one or more input / output (I / O) devices 108, memory 116, storage 114, and / or a network interface 106.The one or more processors 102 may include any suitable processor implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), an artificial intelligence (AI) accelerator, a parallel processing unit (PPU), a data processing unit (DPU), any other type of processing unit, or a combination of different processing units, such as one or more CPUs configured to operate in conjunction with one or more GPUs. In general, the one or more processors 102 may include any technically feasible hardware unit capable of processing data and / or executing software applications.Furthermore, in the context of this disclosure, the computing elements shown in computing device 100 may correspond to a physical computing system (e.g., a system in a data center) and / or may correspond to a virtual computing instance executing in a computing cloud.

[0028] In at least one embodiment, I / O devices 108 include devices capable of receiving input, such as a keyboard, a mouse, a touch screen, a touchpad, a VR / MR / AR headset, a gesture recognition system, and / or a microphone, as well as devices capable of providing output, such as one or more display devices, one or more haptic devices, and / or one or more speakers. In addition, I / O devices 108 may include devices capable of both receiving input and providing output, such as a touch screen, a Universal Serial Bus (USB) port, etc. I / O devices 108 may be configured to receive various types of input from an end user (e.g.,a designer) of the computing device 100 and also provide various types of outputs to the end user of the computing device 100, such as displayed digital images or digital videos or text. In some embodiments, one or more of the I / O devices 108 are configured to couple the computing device 100 to a network 110.

[0029] In one embodiment, network 110 is any technically feasible type of communications network that enables the exchange of data between computing device 100 and internal, local, remote, or external units or devices, such as a web server or other networked computing device. For example, network 110 may include, among others, a wide area network (WAN), a local area network (LAN), a wireless network (e.g., Wi-Fi), a cellular network, and / or the Internet.

[0030] In at least one embodiment, storage 114 includes non-volatile memory for applications and data, and may include fixed or removable drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid-state storage devices. The management engine 122 and / or the verification engine 124 may be stored in storage 114 and loaded into memory 116 upon execution.

[0031] In one embodiment, memory 116 includes a random-access memory (RAM) module, a flash memory device, and / or any other type of memory device, or a combination thereof. The one or more processors 102, the I / O device interface 104, and the network interface 106 may be configured to read and write data to memory 116. Memory 116 may contain various software programs executable by the one or more processors 102, as well as application data associated with said software programs, including the management engine 122 and / or the processing engine 124.

[0032] The management engine 122 and the processing engine 124 include functionality for managing map data 126 stored in memory 116. The memory 116 may be included, for example, in an on-board system on a chip (SoC) of a vehicle. During vehicle operation, map data 126 may be received from the network 110 via the network interface 106 and stored in memory 116 to enable the vehicle to perform localization, planning, navigation, and / or other location-based tasks. As the vehicle moves to new geographic areas, additional map data 126 for the new geographic areas is downloaded and stored in memory 116.Since the memory 116 in the SoC is limited, the additional map data 126 is typically accommodated in the memory 116 by flushing at least some of the existing map data 126 from the memory 116.

[0033] In one or more embodiments, the management engine 122 calculates priority number values for discrete units of map data 126 stored in memory 116, also referred to herein as map data units. These map data units may include (but are not limited to) map tiles, layers within map tiles, and / or other discrete pieces of map data. The priority number values may be calculated based on attributes of the map data units, such as (but are not limited to) recency of use, frequency of use, distance from a current location of the location-aware system, overlap with a frequent or known route of the location-aware system, a map data unit size, a time of day, and / or a day of the week.Each priority number value may represent a probability of use for a corresponding map data unit and / or another measure of a potential “cost” associated with clearing the map data unit.

[0034] The management engine 122 generates a ranking of these map data units by the corresponding priority numerical values and uses the ranking to select one or more map data units as candidates for eviction. When new map data 126 is received via the network 110 and the network interface 106 for storage in the memory 116, the processing engine 124 makes room for the just-downloaded map data 126 by deleting map data units identified by the management engine 122 as candidates for eviction from the memory 116. The operations of the management engine 122 and the processing engine 124 are described in more detail below.

[0035] Fig. Figure 2 is a more detailed illustration of the management engine 122 and the processing engine 124 of Fig. 1 according to various embodiments. As explained herein, management engine 122 and processing engine 124 operate to manage the storage and retrieval of a number of map data units 206(1)-206(N) (each of which is individually referred to herein as a map data unit 206) in memory 116 on a location-aware system.

[0036] In some embodiments, map data units 206 include discrete units of geographically indexed map data that can be used to perform tasks such as (but not limited to) geolocation, mapping, navigation, and / or vehicle planning. For example, map data units 206 may include map tiles that correspond to adjacent rectangles, squares, cells, hexagons, polygons, and / or other shapes into which a map is divided. These map tiles can be associated with multiple "zoom" levels, such that a map tile at a lower zoom level may contain data covering a larger area (e.g., an entire city), while a map tile at a higher zoom level may contain more detailed information for a smaller area (e.g., a single building or city block).The map data units 206 may also, or instead, include individual layers into which the data within a given map tile is organized. Each layer may store a different type of data for the region represented by the map tile, such as (but not limited to) imagery, roads, highways, road markings, traffic signs, radar points, traffic conditions, weather conditions, network conditions, landmarks, topography, and / or metadata.

[0037] As in Fig. 2, the management engine 122 determines a different set of attributes 210(1)-210(N) (each of which is individually referred to herein as attributes 210) for each of the map data units 206(1)-206(N). In some embodiments, each set of attributes 210 includes information that can be used to characterize the past, current, and / or future use of the corresponding map data unit 206 by the location-aware system. For example, the attributes 210 may include (but are not limited to) a last time of use, frequency of use, geographic distance from a current location of the location-aware system, orientation (e.g., overlap, proximity, etc.) with a frequent or known route of the location-aware system, time of day, and / or day of week.One or more attributes 210 may also, or instead, characterize a "cost" associated with downloading, storing, and / or clearing the corresponding map data unit 206. For example, the attributes 210 may include (but are not limited to) a size of a corresponding map data unit 206, a data type (e.g., images, roads, highways, etc.) stored in the corresponding map data unit 206, an importance of the data stored in the corresponding map data unit 206 for one or more tasks (e.g., mapping, navigation, vehicle planning, etc.) performed by the location-aware system, a number of requests for data stored in the corresponding map data unit 206, and / or a download time associated with the corresponding map data unit 206.

[0038] It is understood that management engine 122 may use various techniques to determine attributes 210 for each map data unit 206. For example, management engine 122 may analyze requests, accesses to memory 116, logs, and / or other historical data associated with map data units 206 to identify trends in frequency of use, recency of use, download times, and / or other data related to the downloading, storage, and / or use of each map data unit 206.The management engine 122 may also or instead use system calls and / or configuration data to determine the size of each map data unit 206, the importance of each map data unit 206 to the operation of the location-aware system, the type of data stored in a given map data unit 206, the current location of the location-aware system, and / or the proximity of the location-aware system to regions represented by map data units 206.

[0039] The management engine 122 may also, or instead, use predictive analytics to predict the likelihood of future use and / or a usage pattern for each map data unit 206. These predictive analytics may include machine learning and / or time series analysis techniques to consider and / or determine correlations across times of day, days of the week, seasonal patterns, weather conditions, road conditions, network conditions, and / or other factors that may affect the operation of the location-aware system and / or the ability of the location-aware system to download and / or use map data units 206.

[0040] The management engine 122 uses attributes 210(1)-210(N) for each of the map data units 206(1)-206(N) to calculate priority number values 212(1)-212(N) (each of which is individually referred to herein as a priority number value 212) for individual map data units 206(1)-206(N). Each priority number value 212 represents a probability that the corresponding map data unit 206 will be reused within a particular timeframe, a measure of the overhead associated with downloading the corresponding map data unit 206, and / or another measure of a potential "cost" associated with evicting the corresponding map data unit 206.

[0041] In one or more embodiments, the management engine 122 calculates priority number values 212 using a set of heuristics and / or rules that may be specified and / or defined based on domain knowledge and / or parameters associated with the use of a location-aware system. For example, the management engine 122 may calculate priority number values 212 as numerical positions of the corresponding map data units 206 in one or more rankings 202. In this example, the management engine 122 may generate multiple "buckets" representing different ranges of recency of use of map data units 206 (e.g., ranges of elapsed time since the last use of map data units 206). Within a particular bucket, the management engine 122 may generate a ranking of the corresponding map data units 206 by frequency of use.The priority number value for a given map data unit 206 may then be determined as the position of the bucket in which that map data unit 206 is located, as well as the position of that map data unit 206 within the ranking of map data units 206 in the bucket.

[0042] The management engine 122 also, or instead, calculates each priority numeric value 212 as a weighted combination of the corresponding set of attributes 210. For example, the management engine 122 may assign each attribute a weight that represents the relative importance of the attribute for the process of clearing map data units 206 from memory 116. The weight may be determined by one or more users, regression, and / or other statistical analysis techniques, and / or other methods. Each attribute may be combined (e.g., multiplied) with the corresponding weight to obtain a weighted attribute, and the priority numeric value 212 for a given map data unit 206 may be calculated as an aggregation (e.g., sum, arithmetic mean, geometric mean, etc.) of weighted attributes 210 for that map data unit 206.

[0043] The management engine 122 also or instead uses one or more machine learning models to calculate priority number values 212 from attributes 210 for the corresponding map data units 206. Each machine learning model may include (but is not limited to) a neural network, a support vector machine, a tree-based model, a regression model, a hierarchical model, an ensemble model, a time series analysis model, and / or another type of model capable of performing general-purpose or specialized artificial intelligence-oriented operations.The one or more machine learning models may be trained using historical data relating to the download and use of map data units in one or more location-aware systems, which may include or exclude the location-aware system on which the management engine 122 and the processing engine 124 are running. During training, the parameters of each machine learning model may be updated so that the machine learning model outputs a binary numeric value indicating whether or not a map data unit will be reused within a certain forward-looking period, taking into account inputs containing attributes associated with the map data unit from one or more periods prior to the forward-looking period.After the machine learning model is trained, additional sets of attributes 210 for map data units 206 stored in memory 116 may be input to the machine learning model, and the machine learning model may generate a priority number value ranging between 0 and 1 that indicates the probability that the corresponding map data unit will be used within a certain time frame corresponding to the look-ahead period. The one or more machine learning models may also be retrained (e.g., on a regular and / or continuous basis) using attributes 210, priority number values 212, and / or results associated with map data units 206 to adapt the machine learning model to usage patterns associated with the location-aware system.The one or more newly trained machine learning models may then be used to convert attributes 210 for subsequent map data units 206 into priority number values 212 that better reflect the costs and / or outcomes associated with evicting map data units 206 from the location-aware system's memory 116.

[0044] It is understood that heuristics, weights, and / or machine learning models employed by management engine 122 may be used to generate priority number values 212 representing a variety of values and / or metrics, and that these values and / or metrics may be used to perform evictions 224 of map data units 206 according to the different priorities. For example, management engine 122 may convert a given set of attributes 210 into a priority number value representing a cost associated with re-downloading a corresponding map data unit 206 should that map data unit be evicted and later requested again.Attributes 210 that may contribute to this type of priority score may include (but are not limited to) the size of the map data unit, the current and / or forecasted network bandwidth, and / or estimated and / or historical download times for the map data unit. This cost-based priority score may be used to improve the latency, responsiveness, and / or resource consumption of the location-aware system.

[0045] In another example, the management engine 122 may convert a given set of attributes 210 into a priority numeric value representing the contextual relevance of a corresponding map data item 206 to a current and / or predicted context (e.g., location, route, task, etc.) of the location-aware system. Attributes used to calculate this numeric value may include (but are not limited to) historical route data, planned future routes, traffic conditions, weather conditions, and / or points of interest or frequent destinations within the region represented by that map data item 206. This context-based numeric value may facilitate the retention of map data items 206 that are in the vicinity of the vehicle's trajectory, in areas the vehicle is likely to visit, and / or in areas the vehicle frequently visits.

[0046] In a third example, the management engine 122 may convert a given set of attributes 210 into a priority number value calculated from the age and expected update frequency of the data stored in the corresponding map data unit 206. Therefore, map data units 206 that store frequently changing data (e.g., traffic conditions, construction updates, etc.) may have priority number values 212 that result in a higher probability of clearing than map data units 206 that store less frequently changing data (e.g., roads, traffic signs, landmarks, etc.). This type of priority number value may therefore be used to target map data units that are outdated or obsolete for clearing.

[0047] In a fourth example, the management engine 122 may combine one or more of the above-mentioned types of priority number values into a "composite" priority number value 212 that represents an overall measure of the utility of a corresponding map data unit 206. This composite priority number value 212 may allow the management engine 122 to balance multiple objectives associated with managing the storage and eviction of map data units 206 in memory 116.

[0048] After priority numeric values 212 are calculated for map data units 206, management engine 122 generates one or more rankings 202 of map data units 206 by the corresponding priority numeric values 212. Management engine 122 also uses rankings 202 and / or priority numeric values 212 to select one or more map data units 206 from rankings 202 as eviction candidates 204. For example, management engine 122 may select a prespecified number of map data units 206 with positions in rankings 202 that represent the lowest likelihood of reuse, download cost, contextual relevance, and / or likelihood of being up-to-date as eviction candidates 204.In another example, the management engine 122 may select a prespecified and / or variable number of card data units 206 having priority number values 212 that fall below a threshold as eviction candidates 204. Techniques for determining eviction candidates 204 from priority number values 212 and / or rankings 202 are discussed below with respect to FIG. Fig. 3A and Fig. 3B is described in more detail.

[0049] Management engine 122 transmits eviction candidates 204 to processing engine 124, and processing engine 124 uses eviction candidates 204 to perform evictions 224 of map data units 206 from memory 116. For example, management engine 122 may periodically and / or continuously select and transmit eviction candidates 204 to processing engine 124. When processing engine 124 detects and / or receives downloads 220 of new map data units 216(1)-216(M) (each of which is individually referred to herein as a new map data unit 216), processing engine 124 may perform evictions 224 of one or more map data units 206 from the most recent eviction candidates 204 to make room for the downloaded new map data units 216.These evictions 224 may be performed based on priority number values 212 for map data units 206 in eviction candidates 204, the sizes of map data units 206 in eviction candidates 204, and / or other criteria.

[0050] Fig. 3A illustrates the exemplary operation of the management engine 122 of Fig. 1 in determining a set of eviction candidates 204 according to various embodiments. In the example of Fig. 3A, the management engine 122 operates with map data units 206 that include individual layers 304(1)-304(X) and 304(X*Y-Y+1)-304(X*Y) within a set of map tiles 302(1)-302(Y) (each of which is individually referred to herein as a map tile 302). As mentioned herein, the map tiles 302 may include rectangular, hexagonal, square, and / or other types of discrete regions into which map data is divided. Each map tile 302 may store different types of data (e.g., images, roads, highways, lane markings, traffic signs, radar points, traffic conditions, weather conditions, landmarks, topography, metadata, etc.) in different layers 304.

[0051] More specifically, the management engine 122 determines a set of attributes 210(1)-210(X) and 210(X*Y-Y+1)-210(X*Y) for each of the layers 304(1)-304(X) and 304(X*Y-Y+1)-304(X*Y). For example, the management engine 122 may use attributes 210 such as (but not limited to) a layer size, a layer importance, a number of requests associated with a layer, a staleness of data in a layer, and / or a download time associated with a layer to calculate priority number values 212 representing the cost of clearing the corresponding layers 304.

[0052] The management engine 122 also converts each set of attributes 210(1)-210(X) and 210(X*Y-Y+1)-210(X*Y) into a corresponding priority number value 212(1)-212(X) and 212(X*Y-Y+1)-212(X*Y). The management engine 122 then generates a ranking 202 of the layers 304 according to priority number values 212 and selects eviction candidates 204 from the ranking 202. In this example, the management engine 122 therefore uses the layers 304 in the map tiles 302 as map data units 206 that can be selected as eviction candidates 204.

[0053] Fig. 3B illustrates the exemplary operation of the management engine 122 of Fig. 1 in determining a set of eviction candidates 204 according to various embodiments. In the example of Fig. 3B, the management engine 122 uses the map tiles 302(1)-302(Y) as a first type of map data unit 206 for which the attributes 210(1)-210(Y) and the priority number values 212(1)-212(Y) are determined. For example, the management engine 122 may use attributes 210 such as (but not limited to) a last time of use, a frequency of use, a geographic distance from a current location of the location-aware system, an overlap with a route associated with a vehicle, a map tile size, a time of day, and / or a day of the week to calculate priority number values 212 representing the probabilities of reusing the corresponding map tiles 302.

[0054] The management engine 122 generates a ranking 202(1) of map tiles 302(1)-302(Y) by the corresponding priority numerical values 212(1)-212(Y) and selects one or more map tile eviction candidates 204(A) from the ranking 202(1). Continuing the above example, the management engine 122 may select map tile eviction candidates 204(A) as one or more map tiles 302(1)-302(Y) with priority numerical values 212 that represent the lowest probability of being reused.

[0055] The management engine 122 then uses layers 304(1)-304(Z) in one or more map tiles 302 corresponding to map tile eviction candidates 204(A) as a second type of map data unit 206 for which additional attributes 210(Y+1)-210(Y+Z) and priority number values 212(Y+1)-212(Y+Z) are determined. For example, the management engine 122 may use attributes 210(Y+1)-210(Y+Z) such as (but not limited to) a tier size, a tier importance, a number of requests associated with a tier, an aging of data in a tier, and / or a download time associated with a tier to calculate priority number values 212(Y+1)-212(Y+Z) that represent the cost of clearing the corresponding tiers 304(1)-304(Z).

[0056] The management engine 122 generates another ranking 202(2) of the layers 304(1)-304(Z) by the corresponding priority numerical values 212(Y+1)-212(Y+Z) and selects one or more layer eviction candidates 204(A) from the ranking 202(2). Continuing the above example, the management engine 122 may select layer eviction candidates 204(B) as one or more layers 304(1)-304(Z) with priority numerical values 212(Y+1)-212(Y+Z) that represent the lowest cost for eviction.

[0057] By selecting eviction candidates 204 from individual layers 304 of map tiles 302 in the examples of Fig. 3A and Fig. 3B, the management engine 122 may perform a more fine-grained, intelligent eviction of map data units 206 than naive or unintelligent approaches that evict entire map tiles from memory to make room for newly downloaded map tiles. For example, the examples of Fig. 3A and Fig. 3B allow the management engine 122 to prioritize clearing larger layers, such as images and / or road geometry. The examples of Fig. 3A and Fig. 3B may also or instead allow the management engine 122 to avoid and / or reduce the clearing of smaller, more important layers, such as metadata layers that store manifests, signatures, and / or other information needed to download, verify, and / or use other layers in the corresponding map tiles 302.

[0058] Although the operation of the management machine 122 with respect to Fig. 3A and Fig. 3B as selecting eviction candidates 204 based on map data units 206 that include layers 304 in map tiles 302 and each include both map tiles 302 and layers 304, it should be understood that management engine 122 may select eviction candidates 204 based on other types and / or combinations of map data units 206. For example, management engine 122 may calculate priority number values 212 for individual layers 304 based on the attributes of both those layers 304 and the map tiles 302 in which the layers are located. Management engine 122 may then use these priority number values 212 to select one or more layers 304 and / or one or more map tiles 302 as eviction candidates 204.In another example, the management engine 122 may calculate priority number values 212 and / or select eviction candidates 204 from groupings of map tiles 302 and / or layers 304. In a third example, the management engine 122 may calculate priority number values 212 and / or select eviction candidates 204 from blocks, pages, and / or other units of memory 116 instead of or in addition to calculating priority number values 212 and / or selecting eviction candidates 204 from map tiles 302 and / or layers 304.

[0059] It should be understood that these and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions performed by entities described herein may be performed by hardware, firmware, and / or software. Various functions may, for example, be performed by a processor executing instructions stored in memory.In some embodiments, the systems, methods, and processes described herein may be implemented using similar components, features, and / or functionality as those of the exemplary autonomous vehicle 500 of FIG. Fig. 5A to 5D, the exemplary computing device 600 of Fig. 6 and / or the exemplary data center 700 from Fig. 7.

[0060] With reference to Fig. 4A through 4B, each block of the methods 400 and 430 described herein comprises a computational process that may be performed using any combination of hardware, firmware, and / or software. Various functions may be performed, for example, by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media. The methods may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. Additionally, the methods 400 and 430 are described, by way of example, with respect to the system of Fig. 1. However, these methods may additionally or alternatively be performed by any system or combination of systems, including, but not limited to, the systems described herein.

[0061] Fig. 4A illustrates a flowchart showing a method 400 for managing map data stored in a location-aware system, according to some embodiments of the present disclosure. As shown in Fig. 4A, the method 400 begins with operation 402, in which the management engine 122 determines attributes for a set of map data units stored in memory within a location-aware system. For example, the map data units may include map tiles representing discrete geographic regions into which map data is divided. Attributes associated with the map tiles may include (but are not limited to) a last time of use, a frequency of use, a geographic distance from a current location of the location-aware system, an overlap with a route associated with a vehicle, a map tile size, a time of day, and / or a day of the week. The map data units may also, or instead, include layers within the map tiles.Attributes associated with tiers may include (but are not limited to) a tier size, a tier importance, a number of requests associated with a tier, a tier staleness, and / or a download time associated with a tier.

[0062] In operation 404, the management engine 122 calculates a priority number value for each map data unit based on the corresponding attributes. For example, the management engine 122 may use a set of rules and / or heuristics to generate a priority number value for each map data unit based on some or all of the corresponding attributes. The management engine 122 may also or instead generate a priority number value for each map data unit as a weighted combination of the corresponding attributes. The management engine 122 may also or instead use a machine learning model to convert attributes for a given map data unit into a corresponding priority number value, as described below with respect to Fig. 5. Each priority number value may represent a probability of reusing the corresponding map data unit, a cost associated with re-downloading the map data unit, an amount of memory reclaimed upon clearing the map data unit, and / or another measure associated with a potential consequence of clearing the map data unit.

[0063] In operation 406, the management engine 122 determines one or more map data units to be included in a set of eviction candidates based on the priority numeric values for the map data units. For example, the management engine 122 may generate one or more rankings of the map data units by the priority numeric values. The management engine 122 may also use the ranking(s) and / or priority numeric values to select one or more map data units with priority numeric values that represent the lowest cost of eviction (or the highest benefit of eviction) for inclusion in the set of eviction candidates.

[0064] In operation 408, processing engine 124 determines whether or not one or more new map data units have been received for storage in memory. For example, processing engine 124 may download the one or more new map data units over a network, receive events related to the download of the new map data unit(s), and / or otherwise detect the one or more new map data units.

[0065] If one or more new map data units have been received, the processing engine 124 performs operation 410, wherein the processing engine 124 causes one or more eviction candidates to be evicted from memory. For example, the processing engine 124 may select one or more map data units from the set of eviction candidates to be evicted from memory. The processing engine 124 may also output the one or more selected map data units, generate one or more system calls to delete the one or more selected map data units and / or overwrite the one or more selected map data units with the one or more map data units, and / or perform other operations to effect eviction of the selected map data unit(s) from memory.

[0066] If the processing engine 124 determines that no new map data units have been received and / or performs operation 410 to flush one or more map data units from memory, the management engine 122 performs operation 412, in which the management engine 122 determines whether or not to continue managing the map data. For example, the management engine 122 may determine that map data management will continue while memory is being used to store map data and / or during operation of the location-aware system.

[0067] While the map data continues to be managed, management engine 122 and processing engine 124 repeat operations 402, 404, 406, 408, 410, and 412 on a periodic and / or continuous basis. For example, management engine 122 may perform operations 402, 404, and 406 to generate up-to-date attributes, priority number values, and eviction candidates from the map data units stored in memory. Management engine 122 and processing engine 124 may then perform operations 408, 410, and / or 412 to accommodate new map data units received for storage in memory.Management engine 122 and processing engine 124 may continue to use method 400 to manage the storage and eviction of map data from memory until the memory, map data, and / or location-aware system are no longer in use.

[0068] Fig. 4B illustrates a flowchart showing a method 430 for generating priority number values for map data units using a machine learning model, according to some embodiments of the present disclosure. As shown in Fig. 4A, the method 430 begins with operation 432, where the management engine 122 determines attributes and statistics associated with the use of a set of map data items by one or more location-aware systems. For example, the management engine 122 may collect and / or analyze historical data associated with map data items stored in the one or more location-aware systems. The management engine 122 may determine a set of attributes for a map data item from historical data falling within a first time period and one or more statistics for the map data item from historical data falling within a second time period following the first time period.The attributes may contain values related to the use of the map data unit, data in the map data unit, and / or the operation of the corresponding location-aware system during the first period. The statistic(s) may contain values related to the subsequent use, clearing, and / or re-downloading of the map data unit by the location-aware system during the second period.

[0069] In operation 434, the management engine 122 trains a machine learning model based on the attributes and statistics. For example, the management engine 122 may input a set of attributes for each map data unit into the machine learning model and execute the machine learning model to obtain an output containing predictions of outcomes represented by the corresponding statistics. The management engine 122 may also calculate one or more losses between the predictions and the statistics and use a training technique (e.g., gradient descent and backpropagation) to iteratively update parameters of the machine learning model in a manner that reduces the losses.

[0070] In operation 436, the management engine 122 executes the trained machine learning model to convert additional attributes for an additional set of map data units into corresponding priority number values. For example, the management engine 122 may perform operation 436 to generate priority number values that can be used to selectively evict map data units currently stored in memory on one or more location-aware systems, as explained herein. The additional set of map data units may be stored in the same one or more location-aware systems as the set of map data units used to train the machine learning model in operations 432 and 434.One or more map data units in the additional set of map data units may also or instead be stored in one or more location-aware systems that are different from the one or more location-aware systems from which training data for the machine learning model was obtained.

[0071] In operation 438, the management engine 122 determines whether or not to continue generating priority number values. For example, the management engine 122 may determine that priority number values should continue to be generated while the machine learning model is being used to generate the priority number values, while the priority number values are being used to manage the storage and eviction of map data units in memory on one or more location-aware systems, and / or based on other conditions or criteria. While the management engine 122 determines that priority number values should continue to be generated, the management engine 122 may repeat operations 432 and 434 to retrain the machine learning model using the most recent attributes and statistics from one or more location-aware systems.Management engine 122 may also repeat operations 436 and 438 to use the newly trained machine learning models to generate priority number values for additional map data units on the one or more same location-aware systems and / or different location-aware systems and determine whether or not to continue generating priority number values using the machine learning model. Consequently, management engine 122 may use method 430 to continuously update the machine learning model and the priority number values as new data becomes available and / or the usage patterns of the one or more location-aware systems change.

[0072] The systems and methods described herein may be used, without limitation, by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), guided and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, hydrofoils, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Furthermore, the systems and methods described herein may be used for a variety of purposes, including, without limitation, machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, generative AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or other suitable applications.

[0073] The disclosed embodiments may include a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented with a robot, aviation systems, media systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing light transport simulations,Systems for performing collaborative content creation for 3D assets, systems for performing operations using generative AI, systems that implement one or more language models - such as one or more large language models (LLMs), systems that are implemented at least partially using cloud computing resources, and / or other types of systems. EXEMPLARY AUTONOMOUS VEHICLE

[0074] Fig. 5A is an illustration of an example autonomous vehicle 500, according to some embodiments of the present disclosure. The autonomous vehicle 500 (alternatively referred to herein as "vehicle 500") may include, without limitation, a passenger vehicle, such as a car, a truck, a bus, an emergency service vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire engine, a police vehicle, an ambulance, a boat, a construction vehicle, an underwater vehicle, a robotic vehicle, a drone, an aircraft, a vehicle coupled to a trailer (e.g., a semi-trailer used to transport cargo), and / or another type of vehicle (e.g., that is unmanned and / or accommodates one or more passengers).Autonomous vehicles are generally described in terms of levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) standard "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018, Standard No. 13016-201609, published September 30, 2016, and prior and future versions of this standard). Vehicle 500 may exhibit functionality consistent with one or more of the Level 3 through Level 5 autonomous driving levels.The vehicle 500 may exhibit functionality according to one or more of Level 1 through Level 5 autonomous driving levels. For example, depending on the embodiment, the vehicle 500 may be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term "autonomous" as used herein may include any and / or all types of autonomy for the vehicle 500 or other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, assisted autonomy, semi-autonomous, primarily autonomous, or another designation.

[0075] The vehicle 500 may include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle 500 may include a propulsion system 550, such as an internal combustion engine, a hybrid electric power plant, a pure electric motor, and / or another type of propulsion system. The propulsion system 550 may be connected to a drivetrain of the vehicle 500, which may include a transmission to enable propulsion of the vehicle 500. The propulsion system 550 may be controlled in response to receiving signals from the throttle or accelerator 552.

[0076] A steering system 554, which may include a steering wheel, may be used to steer the vehicle 500 (e.g., along a desired path or route) when the propulsion system 550 is operating (e.g., when the vehicle is moving). The steering system 554 may receive signals from a steering actuator 556. The steering wheel may be optional for full automation (Level 5).

[0077] The brake sensor system 546 may be used to apply the vehicle brakes in response to receiving signals from the brake actuators 548 and / or the brake sensors.

[0078] The one or more controllers 536 that control one or more systems on chips (SoCs) 504 ( Fig. 5C) and / or GPUs may provide signals (e.g., representative of instructions) to one or more components and / or systems of the vehicle 500. For example, the one or more controllers may send signals to apply the vehicle brakes via one or more brake actuators 548, to apply the steering system 554 via one or more steering actuators 556, to apply the propulsion system 550 via one or more throttle / accelerator devices 552. The one or more controllers 536 may include one or more built-in (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 500.The one or more controllers 536 may include a first controller 536 for autonomous driving functions, a second controller 536 for functional safety functions, a third controller 536 for artificial intelligence functions (e.g., computer vision), a fourth controller 536 for infotainment functions, a fifth controller 536 for emergency redundancy, and / or other controllers. In some examples, a single controller 536 may perform two or more of the above functionalities, two or more controllers 536 may perform a single functionality, and / or any combination thereof.

[0079] The one or more controllers 536 may provide the signals to control one or more components and / or systems of the vehicle 500 in response to sensor data received from one or more sensors (e.g., sensor inputs). The sensor data may be received, for example and without limitation, from one or more of the following: global navigation satellite systems (“GNSS”) sensor(s) 558 (e.g., global positioning system sensor(s)), RADAR sensor(s) 560, ultrasonic sensor(s) 562, LIDAR sensor(s) 564, inertial measurement unit (IMU) sensor(s) 566 (e.g., accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s), etc.), microphone(s) 596, stereo camera(s) 568, wide-angle camera(s) 570 (e.g., fisheye cameras), infrared camera(s) 572, ambient camera(s) 574 (e.g.,360-degree cameras), long-range and / or medium-range camera(s) 598, speed sensor(s) 544 (e.g., for measuring the speed of the vehicle 500), vibration sensor(s) 542, steering sensor(s) 540, brake sensor(s) (e.g., as part of the brake sensor system 546), and / or other sensor types.

[0080] One or more of the controllers 536 may receive inputs (e.g., in the form of input data) from an instrument cluster 532 of the vehicle 500 and provide outputs (e.g., in the form of output data, display data, etc.) via a human-machine interface (HMI) display 534, an audible annunciator, a speaker, and / or via other components of the vehicle 500. The outputs may include information such as vehicle speed, RPM, time, map data (e.g., map data 126, map data units 206, the high-definition (“HD”) map 522 of Fig. 5C), location data (e.g., the location of the vehicle 500, e.g., on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the one or more controllers 536, etc. For example, the HMI display 534 may display information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing traffic light, etc.) and / or information about maneuvers that the vehicle has performed, is currently performing, or will perform (e.g., change lanes now, take exit 34B in two miles, etc.).

[0081] The vehicle 500 further includes a network interface 524 that may utilize one or more wireless antennas 526 and / or modems for communication over one or more networks. The network interface 524 may, for example, be capable of communication via Long-Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communications (GSM), IMT-CDMA Multi-Carrier (CDMA2000), etc. The one or more wireless antennas 526 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc.and / or Low Power Wide Area Networks (LPWANs), such as LoRaWAN, SigFox, etc.

[0082] Fig. 5B is an example of camera locations and fields of view for the example autonomous vehicle 500 of Fig. 5A, according to some embodiments of the present disclosure. The cameras and respective fields of view are an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at various locations on the vehicle 500.

[0083] The camera types for the cameras may include, but are not limited to, digital cameras that may be configured for use with the components and / or systems of the vehicle 500. The one or more cameras may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera types may be capable of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The cameras may use rolling shutters, global shutters, another type of shutter, or a combination thereof.In some examples, the color filter array may include a red clear clear clear (RCCC) color filter array, a red clear clear blue (RCCB) color filter array, a red blue green clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor color filter array (RGGB), a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear-pixel cameras, such as cameras with an RCCC, an RCCB, and / or an RBGC color filter array, may be used to increase light sensitivity.

[0084] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multifunction mono camera can be installed to provide features including lane departure warning, traffic sign assist, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0085] One or more of the cameras may be mounted in a bracket, such as a specially designed (three-dimensional ("3D") printed) bracket, to eliminate stray light and reflections from the vehicle interior (e.g., dashboard reflections reflected in the windshield) that could interfere with the camera's image data acquisition. With regard to the bracket for exterior mirrors, the exterior mirrors may be custom 3D printed so that the camera mounting plate is adapted to the shape of the exterior mirror. In some examples, the one or more cameras may be integrated into the exterior mirror. For side-mounted cameras, the one or more cameras may also be integrated into the four pillars at each corner of the cabin.

[0086] Cameras with a field of view that includes portions of the environment in front of the vehicle 500 (e.g., forward-facing cameras) can be used for the surrounding view to help identify forward paths and obstacles, and to provide information critical to establishing an occupancy grid and / or determining preferred vehicle paths with the assistance of one or more controllers 536 and / or control SoCs. Forward-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS features and systems that include lane departure warnings (LDW), autonomous cruise control (ACC), and / or other features such as traffic sign detection.

[0087] A variety of cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform that includes a complementary metal oxide semiconductor (CMOS) color imager. Another example is the wide-angle cameras 570, which can be used to capture objects that enter the field of view from the periphery (e.g., pedestrians, crossing vehicles, or bicycles). Although Fig. 5B illustrates only one wide-angle camera, any number (including zero) of wide-angle cameras 570 may be present on the vehicle 500. Furthermore, any number of long-range cameras 598 (e.g., a long-range stereo camera pair) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The one or more long-range cameras 598 may also be used for object detection and classification, as well as basic object tracking.

[0088] Any number of stereo cameras 568 may also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo cameras 568 may include an integrated control unit comprising a scalable processing unit that can provide a programmable logic ("FPGA") and a multi-core microprocessor with an integrated controller area network ("CAN") or Ethernet interface on a single chip. Such a unit can be used to create a 3D map of the vehicle's surroundings that includes a distance estimate for all points in the image. One or more alternative stereo cameras 568 may include a compact stereo vision sensor that can include two camera lenses (one each on the left and right) and an image processing chip that can measure the distance between the vehicle and the target object and process the generated information (e.g.,metadata) to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 568 may be used in addition to or alternatively to those described here.

[0089] Cameras with a field of view that includes portions of the environment to the side of the vehicle 500 (e.g., side cameras) may be used for the environment view and provide information used to create and update the occupancy grid and to generate collision warnings in the event of a side impact. For example, the one or more environment cameras 574 (e.g., four environment cameras 574, as in Fig. 5B) may be positioned on the vehicle 500. The one or more surround cameras 574 may include one or more wide-angle cameras 570, one or more fisheye cameras, one or more 360-degree cameras, and / or the like. For example, four fisheye cameras may be mounted on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may utilize three surround cameras 574 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround camera.

[0090] Cameras with a field of view that includes portions of the environment behind the vehicle 500 (e.g., rearview cameras) may be used for parking assistance, surround view, rear impact warnings, and occupancy grid creation and updating. A variety of cameras may be used, including, but not limited to, cameras that are also suitable as one or more forward-facing cameras (e.g., one or more long-range and / or medium-range cameras 598, one or more stereo cameras 568, one or more infrared cameras 572, etc.), as described herein.

[0091] Fig. 5C is a block diagram of an example system architecture for the example autonomous vehicle 500 of Fig. 5A, according to some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are set forth as examples only. Other arrangements and elements (e.g., engines, interfaces, functions, arrangements, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as individual or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions performed by entities described herein may be performed by hardware, firmware, and / or software. Various functions may be performed, for example, by a processor executing instructions stored in memory.

[0092] Each of the vehicle’s components, features and systems in Fig. 5C is illustrated as being connected via bus 502. Bus 502 may include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). A CAN may be a network within vehicle 500 that serves to support the control of various features and functions of vehicle 500, such as the application of brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus may be read to determine steering wheel angle, vehicle speed, engine speed (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0093] Although bus 502 is described herein as a CAN bus, this is not intended to be a limitation. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to the CAN bus. Furthermore, while a single wire is used to represent bus 502, this is not intended to be a limitation. For example, there may be any number of buses 502, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use a different protocol. In some examples, two or more buses 502 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 502 may be used for collision avoidance functionality and a second bus 502 may be used for actuation control.In each example, each bus 502 may communicate with one of the components of the vehicle 500, and two or more buses 502 may communicate with the same components. In some examples, each SoC 504, each controller 536, and / or each computer within the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle 500) and may be connected to a common bus, such as the CAN bus.

[0094] The vehicle 500 may include one or more controllers 536 as described herein with reference to Fig. 5A. The one or more controllers 536 may be used for a variety of functions. The one or more controllers 536 may be coupled to any of the various other components and systems of the vehicle 500 and may be used for control of the vehicle 500, artificial intelligence of the vehicle 500, infotainment for the vehicle 500, and / or the like.

[0095] The vehicle 500 may include one or more systems on a chip (SoC) 504. The SoC 504 may include one or more CPUs 506, one or more GPUs 508, one or more processors 510, one or more caches 512, one or more accelerators 514, one or more memory 516, and / or other components and features not illustrated. The one or more SoCs 504 may be used to control the vehicle 500 in a variety of platforms and systems. For example, the one or more SoCs 504 may be combined in a system (e.g., the system of the vehicle 500) with an HD card 522 that may be accessed via a network interface 524 from one or more servers (e.g., the one or more servers 578 of Fig. 5D) Receive map refreshes and / or updates. As explained herein, these map refreshes and / or updates may be managed by selectively evicting map tiles, layers, and / or other map data units from the one or more caches 512, the one or more data stores 516, and / or other types of storage or memory in SoC 504 based on attributes and / or priority number values associated with the map data units.

[0096] The one or more CPUs 506 may include a CPU cluster or CPU complex (alternatively referred to herein as a "CCPLEX"). The one or more CPUs 506 may include multiple cores and / or L2 caches. For example, in some embodiments, the one or more CPUs 506 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the one or more CPUs 506 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The one or more CPUs 506 (e.g., the CCPLEX) may be configured to support concurrent operation of clusters, such that any combination of the clusters of the one or more CPUs 506 may be active at any given time.

[0097] The one or more CPUs 506 may implement power management features that include one or more of the following: individual hardware blocks may be automatically clocked when idle to conserve dynamic power; each core clock may be controlled when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power controlled; each core cluster may be independently clock controlled if all cores are clock controlled or power controlled; and / or each core cluster may be independently power controlled if all cores are power controlled. The one or more CPUs 506 may further implement an enhanced power state management algorithm in which allowable power states and expected wake-up times are established, and the hardware / microcode determines the best power state to enter for the core, cluster, and CCPLEX.The processing cores can support simplified sequences for inputting the energy state into the software, offloading the work to the microcode.

[0098] The one or more GPUs 508 may include an integrated GPU (alternatively referred to herein as an "IGPU"). The one or more GPUs 508 may be programmable and may be efficient for parallel workloads. The one or more GPUs 508 may, in some examples, utilize an enhanced Tensor instruction set. The one or more GPUs 508 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of memory capacity), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512 KB of memory capacity). In some embodiments, the one or more GPUs 508 may include at least eight streaming microprocessors. The one or more GPUs 508 may utilize one or more application programming interfaces (APIs) for computations.Additionally, the one or more GPUs 508 may utilize one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0099] The one or more GPUs 508 may be power-optimized for best performance in automotive and embedded use cases. For example, the one or more GPUs 508 may be fabricated on a fin field-effect transistor (FinFET). However, this is not a limitation, and the one or more GPUs 508 may also be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor may include a number of mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block can be assigned 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA TENSOR COREs with mixed precision for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64 KB register file.In addition, the streaming microprocessors can include independent parallel integer and floating-point datapaths to enable efficient execution of workloads with a mix of computations and addressing calculations. The streaming microprocessors can include independent thread scheduling to enable fine-grained synchronization and cooperation between parallel threads. The streaming microprocessors can include a combined L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0100] The one or more GPUs 508 may include high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem to provide, in some examples, a peak memory bandwidth of approximately 900 GB / second. In some examples, synchronous graphics random-access memory (SGRAM), such as double data rate type five (GDDR5), may be used in addition to or alternatively to the HBM memory.

[0101] The one or more GPUs 508 may include unified memory technology that includes access counters to enable more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for processor-shared memory areas. In some examples, address translation services (ATS) support may be used to allow the one or more GPUs 508 to directly access the page tables of the one or more CPUs 506. In such examples, if the memory management unit (MMU) of the one or more GPUs 508 fails, an address translation request may be sent to the one or more CPUs 506.In response, the one or more CPUs 506 may look up the virtual-to-physical mapping for the address in their page tables and send the translation back to the one or more GPUs 508. Thus, the unified memory technology may enable a single unified virtual address space for the memory of both the one or more CPUs 506 and the one or more GPUs 508, thereby simplifying the programming of the one or more GPUs 508 and the porting of applications to the one or more GPUs 508.

[0102] Additionally, the one or more GPUs 508 may include an access counter that can track the frequency of access by the one or more GPUs 508 to memory of other processors. The access counter can help move memory pages to the physical memory of the processor that accesses the pages most frequently.

[0103] The one or more SoCs 504 may include any number of caches 512, including those described herein. For example, the one or more caches 512 may include an L3 cache available to both the one or more CPUs 506 and the one or more GPUs 508 (e.g., connected to both the one or more CPUs 506 and the one or more GPUs 508). The one or more caches 512 may include a write-back cache that may track line states, e.g., by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may also be used.

[0104] The one or more SoCs 504 may include one or more arithmetic logic units (ALUs) that may be utilized in performing processing related to any of the various tasks or operations of the vehicle 500, such as processing DNNs. Additionally, the one or more SoCs 504 may include one or more floating-point units (FPUs)—or other mathematical co-processors or numerical co-processors—for performing mathematical operations within the system. For example, the one or more SoCs 504 may include one or more FPUs integrated as execution units within one or more CPUs 506 and / or one or more GPUs 508.

[0105] The one or more SoCs 504 may include one or more accelerators 514 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the one or more SoCs 504 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the one or more GPUs 508 and offload some of the tasks of the one or more GPUs 508 (e.g., to free up more cycles of the one or more GPUs 508 to perform other tasks). For example, the one or more accelerators 514 may be configured for targeted workloads (e.g.,Perception, convolutional neural networks (CNNs), etc.) that are robust enough to be suitable for acceleration can be used. The term "CNN" as used here can include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0106] The one or more accelerators 514 (e.g., the hardware acceleration cluster) may include a deep learning accelerator (DLA). The one or more DLAs may include one or more tensor processing units (TPUs) configured to provide an additional tens of trillion operations per second for deep learning applications and inferencing. The TPUs may be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). The one or more DLAs may also be optimized for a specific set of neural network types and floating-point operations, as well as for inferencing. The design of the one or more DLAs may provide more performance per millimeter than a general-purpose GPU, far exceeding the performance of a CPU.The one or more TPUs can perform multiple functions, including a single-instance convolution function that supports, for example, INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0107] The one or more DLAs can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for a variety of functions, including, for example and without limitation: a CNN for object identification and detection using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety and / or security events.

[0108] The one or more DLAs can perform any function of the one or more GPUs 508, and by using an inference accelerator, for example, a developer can dedicate either the one or more DLAs or the one or more GPUs 508 to each function. For example, the developer can focus the processing of CNNs and floating-point operations on the one or more DLAs and leave other functions to the one or more GPUs 508 and / or other accelerators 514.

[0109] The one or more accelerators 514 (e.g., the hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may also be referred to herein as a computer vision accelerator. The one or more PVAs may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The one or more PVAs may provide a balance between performance and flexibility. For example, and without limitation, each PVA may include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA) cores, and / or any number of vector processors.

[0110] The RISC cores may interact with image sensors (e.g., the image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of the RISC cores may include any amount of memory. The RISC cores may use any number of protocols, depending on the embodiment. In some examples, the RISC cores may execute a real-time operating system (RTOS). The RISC cores may be implemented with one or more integrated circuits, application-specific integrated circuits (ASICs), and / or memory devices. The RISC cores may include, for example, an instruction cache and / or tightly coupled RAM.

[0111] The DMA may enable components of the PVA(s) to access the system's memory independently of the one or more CPUs 506. The DMA may support any number of features designed to optimize the PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In some examples, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0112] The vector processors may be programmable processors that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing functions. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or memory (e.g., VMEM).A VPU core can contain a digital signal processor, such as a single instruction multiple data (SIMD) and very long instruction word (VLIW) digital signal processor. Combining SIMD and VLIW can increase throughput and speed.

[0113] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each of the vector processors may be configured to operate independently of the other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, the multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may concurrently execute different computer vision algorithms on the same image, or even execute different algorithms on consecutive images or portions of an image.Among other things, any number of PVAs can be included in the hardware acceleration cluster, and any number of vector processors can be included in each of the PVAs. Furthermore, the one or more PVAs can contain additional memory for error correcting code (ECC) to increase the overall security of the system.

[0114] The one or more accelerators 514 (e.g., the hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM to the one or more accelerators 514. In some examples, the on-chip memory may include at least 4 MB of SRAM, consisting of, for example, and without limitation, eight field-configurable memory blocks accessible by both the PVA and the DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and the DLA may access the memory through a backbone that provides high-speed access to the memory to the PVA and the DLA.The backbone may include an on-chip computer vision network connecting the PVA and DLA to the memory (e.g., using the APB).

[0115] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA are delivering ready and valid signals before transmitting control signals / addresses / data. Such an interface can provide separate phases and channels for transmitting control signals / addresses / data, as well as bursty communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508, although other standards and protocols can also be used.

[0116] In some examples, the one or more SoCs 504 may include a real-time ray tracing hardware accelerator, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the positions and extents of objects (e.g., within a world model) to generate real-time visualization simulations, for radar signal interpretation, for sound propagation synthesis and / or analysis, for simulation of sonar systems, for general wave propagation simulation, for comparison with lidar data for localization, and / or for other functions, and / or for other purposes. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more operations related to ray tracing.

[0117] The one or more accelerators 514 (e.g., the hardware accelerator cluster) have a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for critical processing steps in ADAS and autonomous vehicles. The capabilities of the PVA are well suited to algorithmic areas that require predictable processing with low power and low latency. In other words, the PVA is well suited for semi-dense or dense regular computations, even on small datasets, that require predictable runtimes with low latency and low power. In the context of autonomous vehicle platforms, the PVAs are therefore designed to execute classical computer vision algorithms because they are efficient at object detection and operate on integer mathematics.

[0118] According to one embodiment of the technology, the PVA is used, for example, to perform computer stereovision. In some examples, a semi-global matching-based algorithm may be used, although this is not intended as a limitation. Many applications for Level 3-5 autonomous driving require on-the-fly motion estimation or stereo matching (e.g., structure from motion, pedestrian detection, lane detection, etc.). The PVA can perform a computer stereovision function on inputs from two monocular cameras.

[0119] In some examples, the PVA can be used to perform dense optical flow, such as processing raw radar data (e.g., using a 4D Fast Fourier Transform) to provide processed radar. In other examples, the PVA is used for time-of-flight depth processing, such as processing raw time-of-flight data to provide processed time-of-flight data.

[0120] The DLA can be used to power any type of network to improve control and driving safety; for example, this includes a neural network that outputs a confidence measure for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative "weight" to each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only those detections that exceed the threshold as true positives.In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most certain detections should be considered as triggers for AEB. The DLA may employ a neural network to regress the confidence value. The neural network may take as input at least a subset of parameters, such as, but not limited to, the bounding box dimensions, the ground plane estimate obtained (e.g., from another subsystem), the output of the inertial measurement unit (IMU) sensor 566 correlated with the orientation of the vehicle 500, distance, 3D position estimates of the object obtained by the neural network and / or other sensors (e.g., one or more LIDAR sensors 564 or one or more RADAR sensors 560).

[0121] The one or more SoCs 504 may include one or more data stores 516 (e.g., memory). The one or more data stores 516 may be on-chip memory on the one or more SoCs 504 in which neural networks to be executed on the GPU and / or the DLA may be stored. In some examples, the one or more data stores 516 may be large enough to store multiple instances of neural networks for redundancy and security. The one or more data stores 512 may include one or more L2 or L3 caches 512. The reference to the one or more data stores 516 may include a reference to the memory associated with the PVA, the DLA, and / or one or more other accelerators 514, as described herein.

[0122] The one or more SoCs 504 may include one or more processors 510 (e.g., embedded processors). The one or more processors 510 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle boot power and management functions and associated security enforcement. The boot and power management processor may be part of the boot sequence of the one or more SoCs 504 and may provide runtime power management services. The boot and power management processor may provide clock and voltage programming, assisting with system transitions to a low-power state, managing the thermals and temperature sensors of the one or more SoCs 504, and / or managing the one or more SoCs 504 power states.Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the one or more SoCs 504 may use the ring oscillators to sense the temperatures of the one or more CPUs 506, the one or more GPUs 508, and / or the one or more accelerators 514. If it is determined that the temperatures exceed a threshold, the boot and power management processor may enter a temperature fault routine and place the one or more SoCs 504 into a lower power state and / or place the vehicle 500 into a chauffeur-to-safe-stop mode (e.g., bring the vehicle 500 to a safe stop).

[0123] The one or more processors 510 may also include a set of embedded processors that can serve as an audio processing engine. The audio processing engine may be an audio subsystem that enables full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0124] The one or more processors 510 may also include an always-on processor engine that can provide the necessary hardware functions to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0125] The one or more processors 510 may also include a safety cluster engine containing a dedicated processor subsystem for safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, an interrupt controller, etc.), and / or routing logic. In a safety mode, the two or more cores may operate in a lockstep mode, functioning as a single core with comparison logic that captures any differences between their operations.

[0126] The one or more processors 510 may also include a real-time camera engine, which may include a dedicated processor subsystem for managing the real-time camera.

[0127] The one or more processors 510 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0128] The one or more processors 510 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the player window. The video image compositor may perform lens distortion correction on the one or more wide-angle cameras 570, the one or more surround cameras 574, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensor is preferably monitored by a neural network running on another instance of the enhanced SoC and configured to detect events in the cabin and respond accordingly.An in-cabin system can perform lip reading to activate cellular service and make a call, dictate emails, change the destination, activate or change the vehicle's infotainment system and settings, or enable voice-activated web browsing. Certain functions are available to the driver only when the vehicle is operating in autonomous mode and are disabled otherwise.

[0129] The video compositor can incorporate enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in a video, the noise reduction weights the spatial information accordingly, reducing the weight of information provided by neighboring frames. If a frame or section of a frame contains no motion, the temporal noise reduction performed by the video compositor can use information from the previous frame to reduce noise in the current frame.

[0130] The video image compositor may also be configured to perform stereo distortion correction on the input stereo lens images. The video image compositor may also be used for user interface design when the operating system desktop is in use and the one or more GPUs 508 do not need to constantly render new surfaces. Even when the one or more GPUs 508 are powered on and actively performing 3D rendering, the video image compositor may be used to offload the one or more GPUs 508, thereby improving performance and responsiveness.

[0131] The one or more SoCs 504 may also include a Mobile Industry Processor Interface (MIPI) serial camera interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that may be used for camera and related pixel input functions. The one or more SoCs 504 may also include one or more input / output controllers, one or more of which may be software-controlled and used to receive I / O signals that are not assigned to a specific role.

[0132] The one or more SoCs 504 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The one or more SoCs 504 may be used to process data from cameras (e.g., via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., one or more LIDAR sensors 564, one or more RADAR sensors 560, etc., which may be connected via Ethernet), data from bus 502 (e.g., speed of vehicle 500, steering wheel position, etc.), and data from one or more GNSS sensors 558 (e.g., connected via Ethernet or CAN bus).The one or more SoCs 504 may further include dedicated high-performance mass storage controllers, which may include their own DMA engines and which may be used to offload routine data management tasks from the one or more CPUs 506.

[0133] The one or more SoCs 504 may be an end-to-end platform with a flexible architecture spanning automation levels 3-5, thereby providing a comprehensive functional safety architecture that supports and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The one or more SoCs 504 may be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the one or more accelerators 514, in combination with the one or more CPUs 506, the one or more GPUs 508, and the one or more data memories 516, may form a fast, efficient platform for Level 3-5 autonomous vehicles.

[0134] The technology thus offers capabilities and functions that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using a high-level programming language, such as the C programming language, to execute a variety of processing algorithms on a wide variety of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, such as execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a prerequisite for in-vehicle ADAS applications and a requirement for practical Level 3-5 autonomous vehicles.

[0135] In contrast to conventional systems, the technology described herein enables the simultaneous and / or sequential execution of multiple neural networks and the combination of the results to enable Level 3-5 autonomous driving functionality by providing a CPU complex, a GPU complex, and a hardware acceleration cluster. For example, a CNN running on the DLA or dGPU (e.g., the one or more GPUs 520) may include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing semantic understanding of the sign, and passing this semantic understanding to the path planning modules running on the CPU complex.

[0136] Another example is that multiple neural networks can run simultaneously, as required for driving at Level 3, 4, or 5. For example, a warning sign reading "Caution: Flashing lights indicate black ice," along with an electric light, can be interpreted independently or jointly by multiple neural networks. The sign itself can be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing lights indicate black ice" can be interpreted by a second deployed neural network, which informs the vehicle's path-planning software (preferably running on the CPU complex) that if flashing lights are detected, black ice is present.The turn signal can be identified across multiple images by a third neural network, which informs the vehicle's path planning software of the presence (or absence) of turn signals. All three neural networks can run simultaneously, e.g., within the DLA and / or on the one or more GPUs 508.

[0137] In some examples, a facial recognition and vehicle owner identification CNN may use data from camera sensors to identify the presence of an authorized driver and / or owner of the vehicle 500. The always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and to disable the vehicle in security mode when the owner exits the vehicle. In this way, the one or more SoCs 504 provide security against theft and / or carjacking.

[0138] In another example, a CNN for detecting and identifying emergency vehicles may use data from microphones 596 to detect and identify emergency vehicle sirens. Unlike conventional systems that use general classifiers to detect sirens and manually extract features, the one or more SoCs 504 use the CNN to classify environmental and urban sounds, as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to detect the relative approach speed of the emergency vehicle (e.g., by using the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by one or more GNSS sensors 558.For example, if the CNN is operating in Europe, it will attempt to detect European sirens, and if it is operating in the United States, the CNN will attempt to identify only North American sirens. Once an emergency vehicle is detected, a controller can be used to execute an emergency vehicle safety routine, slow the vehicle, pull over to the side of the road, park the vehicle, and / or idle the vehicle, using ultrasonic sensors 562, until the one or more emergency vehicles pass by.

[0139] The vehicle may include one or more CPUs 518 (e.g., one or more discrete CPUs or one or more dCPUs) that may be coupled to the one or more SoCs 504 via a high-speed connection (e.g., PCIe). The CPUs 518 may include, for example, an x86 processor. The CPUs 518 may be used, for example, to perform a variety of functions, including reconciling potentially inconsistent results between ADAS sensors and the one or more SoCs 504 and / or monitoring the status and health of the one or more controllers 536 and / or the infotainment SoC 530.

[0140] The vehicle 500 may include one or more GPUs 520 (e.g., one or more discrete GPUs or one or more dGPUs) that may be coupled to the one or more SoCs 504 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The one or more GPUs 520 may provide additional artificial intelligence capabilities, e.g., by executing redundant and / or distinct neural networks, and may be used to train and / or update neural networks based on inputs (e.g., sensor data) from sensors of the vehicle 500.

[0141] The vehicle 500 may further include the network interface 524, which may include one or more wireless antennas 526 (e.g., one or more wireless antennas for various communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 524 may be used to enable a wireless connection over the internet to the cloud (e.g., to the one or more servers 578 and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct connection between the two vehicles and / or an indirect connection may be established (e.g., via networks and the internet). Direct connections may be established via vehicle-to-vehicle communication.Vehicle-to-vehicle communication may provide vehicle 500 with information about vehicles in the vicinity of vehicle 500 (e.g., vehicles in front of, beside, and / or behind vehicle 500). This functionality may be part of a cooperative adaptive cruise control feature of vehicle 500.

[0142] The network interface 524 may include an SoC that provides modulation and demodulation functions and enables the one and more controllers 536 to communicate over wireless networks. The network interface 524 may include a radio frequency front end for upconverting from baseband to radio frequency and downconverting from radio frequency to baseband. The frequency conversions may be performed using known methods and / or super-heterodyne methods. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0143] The vehicle 500 may further include one or more data stores 528, which may be located off-chip (e.g., outside the SoCs 504). The one or more data stores 528 may include one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard drives, and / or other components and / or devices capable of storing at least one bit of data.

[0144] The vehicle 500 may further include one or more GNSS sensors 558. The one or more GNSS sensors 558 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) assist in mapping, sensing, occupancy grid creation, and / or path planning. Any number of GNSS sensors 558 may be used, including, for example, and without limitation, a GPS using a USB port with an Ethernet-to-serial (RS-232) bridge.

[0145] The vehicle 500 may further include one or more RADAR sensors 560. The one or more RADAR sensors 560 may be used by the vehicle 500 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The functional safety level of the RADAR may be ASIL B. The one or more RADAR sensors 560 may utilize the CAN and / or bus 502 (e.g., to transmit the data generated by the one or more RADAR sensors 560) for control and access to object tracking data, with access to the raw data occurring over Ethernet in some examples. A variety of RADAR sensor types may be used. The one or more RADAR sensors 560 may be suitable for, for example, front-, rear-, and side-facing RADAR, without limitation. In some examples, one or more Pulse Doppler RADAR sensors are used.

[0146] The one or more RADAR sensors 560 may include various configurations, such as long range with a narrow field of view, short range with a wide field of view, short range side coverage, etc. In some examples, long range RADAR may be used for the adaptive cruise control function. Long range RADAR systems may provide a wide field of view realized through two or more independent scans, such as at a range of 250 m. The one or more RADAR sensors 560 may assist in distinguishing between static and moving objects and may be used by ADAS systems for emergency braking and forward collision warning. Long range RADAR sensors may include a monostatic multi-modal RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface.In a six-antenna example, the middle four antennas can create a focused beam pattern designed to detect the surroundings of vehicle 500 at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can expand the field of view so that vehicles entering or exiting the lane of vehicle 500 can be quickly detected.

[0147] Medium-range radar systems, for example, can have a range of up to 560 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 550 degrees (rear). Short-range radar systems can include, among other features, radar sensors designed for installation at both ends of the rear bumper. When installed at both ends of the rear bumper, such a radar sensor system can generate two beams that continuously monitor the blind spot area to the rear and side of the vehicle.

[0148] Short-range radar systems can be used in an ADAS system to monitor blind spots and / or assist with lane change.

[0149] The vehicle 500 may also include one or more ultrasonic sensors 562. The one or more ultrasonic sensors 562, which may be mounted on the front, rear, and / or sides of the vehicle 500, may be used for parking assistance and / or for creating and updating an occupancy grid. A plurality of ultrasonic sensors 562 may be used, and different ultrasonic sensors 562 may be used for different detection ranges (e.g., 2.5 m, 4 m). The one or more ultrasonic sensors 562 may operate at functional safety levels of ASIL B.

[0150] The vehicle 500 may include one or more LIDAR sensors 564. The one or more LIDAR sensors 564 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The one or more LIDAR sensors 564 may be ASIL B functional safety rated. In some examples, the vehicle 500 may include multiple LIDAR sensors 564 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to deliver data to a Gigabit Ethernet switch).

[0151] In some examples, the one or more LIDAR sensors 564 may be capable of providing a list of objects and their distances for a 360-degree field of view. For example, commercially available LIDAR sensors 564 may have an advertised range of approximately 500 m, with an accuracy of 2 cm to 3 cm, and support for a 500 Mbps Ethernet connection. In some examples, one or more non-prominent LIDAR sensors 564 may be used. In such examples, the one or more LIDAR sensors 564 may be implemented as a small device that may be embedded in the front, rear, sides, and / or corners of the vehicle 500. In such examples, the one or more LIDAR sensors 564 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m, even for objects with low reflectivity.The one or more front-mounted LIDAR sensors 564 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0152] In some examples, LIDAR technologies, such as 3D flash LIDAR, may also be used. 3D flash LIDAR uses a laser flash as the transmitting source to illuminate the vehicle's surroundings up to approximately 200 m. A flash LIDAR unit contains a receptor that records the time of flight of the laser pulse and the reflected light at each pixel, which in turn corresponds to the distance between the vehicle and the objects. Flash LIDAR can enable highly accurate and distortion-free images of the surroundings to be created with each laser flash. In some examples, four flash LIDAR sensors may be deployed, one on each side of the vehicle. Available 3D flash LIDAR systems incorporate a solid-state 3D focal plane array LIDAR camera that contains no moving parts other than a fan (e.g., a non-scanning LIDAR device).The flash lidar device can use a 5-nanosecond pulse of a Class I (eye-safe) laser per frame and collect the reflected laser light as 3D range point clouds and co-registered intensity data. By using flash lidar, and because flash lidar is a solid-state device with no moving parts, the one or more lidar sensors 564 can be less susceptible to motion blur, vibration, and / or shock.

[0153] The vehicle may also include one or more IMU sensors 566. The one or more IMU sensors 566 may, in some examples, be located at the center of the rear axle of the vehicle 500. The one or more IMU sensors 566 may, for example and without limitation, include one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other types of sensors. In some examples, such as in six-axis applications, the one or more IMU sensors 566 may include accelerometers and gyroscopes, while in nine-axis applications, the one or more IMU sensors 566 may include accelerometers, gyroscopes, and magnetometers.

[0154] In some embodiments, the one or more IMU sensors 566 may be implemented as a miniaturized, high-performance GPS-aided inertial navigation system (GPS / INS) that combines microelectromechanical system (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and attitude. Thus, in some examples, the one or more IMU sensors 566 may enable the vehicle 500 to estimate heading without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS with the one or more IMU sensors 566. In some examples, the one or more IMU sensors 566 and the one or more GNSS sensors 558 may be combined into a single integrated unit.

[0155] The vehicle may include one or more microphones 596 mounted in and / or around the vehicle 500. The one or more microphones 596 may be used, among other things, to detect and identify emergency vehicles.

[0156] The vehicle may further include any number of camera types, including one or more stereo cameras 568, one or more wide-angle cameras 570, one or more infrared cameras 572, one or more surround cameras 574, one or more long-range and / or medium-range cameras 598, and / or other camera types. The cameras may be used to capture image data around the entire periphery of the vehicle 500. The types of cameras used depend on the embodiments and requirements for the vehicle 500, and any combination of camera types may be used to provide the necessary coverage around the vehicle 500. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or a different number of cameras.The cameras may support, by way of example and without limitation, Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the one or more cameras is referred to herein with reference to . Fig. 5A and Fig. 5B is described in more detail.

[0157] The vehicle 500 may further include one or more vibration sensors 542. The one or more vibration sensors 542 may measure vibrations from components of the vehicle, such as the one or more axles. For example, changes in vibrations may indicate a change in the road surface. In another example, when two or more vibration sensors 542 are used, the differences between the vibrations may be used to determine friction or slippage on the road surface (e.g., when the difference in vibration is between a driven axle and a free-spinning axle).

[0158] The vehicle 500 may include an ADAS system 538. The ADAS system 538 may include an SoC in some examples. The ADAS system 538 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (CWS), lane centering (LC), and / or other features and functions.

[0159] The ACC systems may use one or more RADAR sensors 560, one or more LIDAR sensors 564, and / or one or more cameras. The ACC systems may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle 500 and automatically adjusts the vehicle speed to maintain a safe distance from preceding vehicles. Lateral ACC performs follow-through and advises the vehicle 500 to change lanes if necessary. Lateral ACC is associated with other ADAS applications, such as LCA and CWS.

[0160] The CACC utilizes information from other vehicles, which may be received via the network interface 524 and / or the one or more wireless antennas 526 from other vehicles over a wireless connection or indirectly via a network connection (e.g., over the Internet). Direct connections may be provided via a vehicle-to-vehicle (V2V) communication link, while indirect connections may be an infrastructure-to-vehicle (I2V) communication link. In general, the V2V communication concept provides information about the immediately preceding vehicles (e.g., vehicles immediately in front of the vehicle 500 and in the same lane as it), while the I2V communication concept provides information about traffic further ahead. CACC systems may include both I2V and V2V information sources.Given the information about the vehicles ahead of vehicle 500, CACC can be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

[0161] FCW systems are designed to warn the driver of a hazard so they can take corrective action. FCW systems utilize a forward-facing camera and / or one or more radar sensors 560 coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component. FCW systems can provide a warning, such as a sound, a visual warning, a vibration, and / or a rapid braking pulse.

[0162] AEB systems detect an impending forward collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. AEB systems can use one or more forward-facing cameras and / or one or more radar sensors 560 coupled with a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision; if the driver does not take corrective action, the AEB system can automatically apply the brakes to prevent or at least mitigate the effects of the predicted collision. AEB systems can incorporate techniques such as dynamic brake support and / or impending crash braking.

[0163] LDW systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 500 crosses lane markings. An LDW system will not activate if the driver indicates an intentional lane departure by activating a turn signal. LDW systems may utilize forward-facing cameras coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component.

[0164] LKA systems are a variant of LDW systems. LKA systems provide steering inputs or braking to correct the vehicle 500 when the vehicle 500 begins to depart from its lane.

[0165] BSW systems detect and warn the driver of vehicles in the car's blind spot. BSW systems may provide a visual, audible, and / or tactile warning signal to indicate that merging into or changing lanes is unsafe. The system may provide an additional warning when the driver activates a turn signal. BSW systems may utilize one or more rear-facing cameras and / or radar sensors 560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component.

[0166] RCTW systems can provide visual, audible, and / or tactile notification when an object outside the range of the rearview camera is detected when the vehicle 500 is reversing. Some RCTW systems include AEB to ensure the vehicle brakes are applied to avoid a crash. RCTW systems can utilize one or more rear-facing RADAR sensors 560 coupled with a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component.

[0167] Conventional ADAS systems can produce false positives, which can be annoying and distracting for the driver, but are typically not catastrophic because ADAS systems warn the driver and give them the opportunity to decide whether a safety issue truly exists and act accordingly. However, in an autonomous vehicle 500, in the event of conflicting results, the vehicle 500 must decide for itself whether to consider the result of a primary computer or a secondary computer (e.g., a first controller 536 or a second controller 536). In some embodiments, the ADAS system 538 may, for example, be a backup and / or secondary computer that provides perception information to a rationality module of the backup computer.The backup computer rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. The outputs of the ADAS system 538 can be provided to a supervising MCU. If the outputs of the primary computer and the secondary computer conflict, the supervising MCU must determine how to resolve the conflict to ensure safe operation.

[0168] In some examples, the primary computer may be configured to provide the monitoring MCU with a confidence value indicating the primary computer's confidence in the chosen outcome. If the confidence value exceeds a threshold, the monitoring MCU may follow the primary computer's instruction regardless of whether the secondary computer provides a conflicting or inconsistent result. If the confidence value does not meet the threshold and the primary and secondary computers indicate different results (e.g., a conflict), the monitoring MCU may arbitrate between the computers to determine the appropriate outcome.

[0169] The monitoring MCU may be configured to run one or more neural networks trained and configured to determine the conditions under which the secondary computer triggers false alarms based on the output from the primary and secondary computers. This allows the one or more neural networks in the monitoring MCU to learn when the output of the secondary computer can and cannot be trusted. For example, if the secondary computer is a radar-based FCW system, a neural network in the monitoring MCU can learn when the FCW system identifies metallic objects that are not actually hazardous, such as a drain grate or manhole cover, which triggers an alarm.Similarly, if the secondary computer is a camera-based LDW system, a neural network in the monitoring MCU can learn to override the LDW system when cyclists or pedestrians are present and lane departure is indeed the safest maneuver. In embodiments including one or more neural networks running on the monitoring MCU, the monitoring MCU can include at least one DLA or GPU suitable for executing the one or more neural networks with associated memory. In preferred embodiments, the monitoring MCU can comprise and / or be included as a component of the one or more SoCs 504.

[0170] In other examples, the ADAS system 538 may include a secondary computer that executes the ADAS functionality according to classic computer vision rules. Thus, the secondary computer may use classic computer vision rules (if-then), and the presence of one or more neural networks in the supervising MCU may improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially against errors caused by software (or software-hardware interfaces).For example, if a software bug or error occurs in the software on the primary computer and the non-identical software code on the secondary computer produces the same overall result, the monitoring MCU can have greater confidence that the overall result is correct and the bug in the software or hardware on the primary computer does not cause a significant error.

[0171] In some examples, the output of ADAS system 538 may be fed into the perception block of the primary computer and / or the dynamic driving task block of the primary computer. For example, if ADAS system 538 displays a forward collision warning due to an object immediately in front of the vehicle, the perception block may use this information in identifying objects. In other examples, the secondary computer may have its own neural network trained to reduce the risk of false positives, as described herein.

[0172] The vehicle 500 may further include the infotainment SoC 530 (e.g., an in-vehicle infotainment (IVI) system). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 530 may include a combination of hardware and software that can be used to provide the vehicle 500 with audio (e.g., music, a personal digital assistant, navigation directions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, rear parking assist, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fuel level, oil level, door open / close, air filter information, etc.).The infotainment SoC 530 may include, for example, radios, record players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free calling, a head-up display (HUD), an HMI display 534, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. The infotainment SoC 530 may further be used to provide information (e.g., visual and / or audible) to one or more users of the vehicle, such as information from the ADAS system 538, autonomous driving information such as planned vehicle maneuvers, road layouts, environmental information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0173] The infotainment SoC 530 may include GPU functionality. The infotainment SoC 530 may communicate with other devices, systems, and / or components of the vehicle 500 via the bus 502 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 530 may be coupled to a supervisory MCU so that the infotainment system's GPU may perform some self-driving functions if the one or more primary controllers 536 (e.g., the primary and / or backup computers of the vehicle 500) fail. In such an example, the infotainment SoC 530 may place the vehicle 500 into a chauffeur-to-safe-stop mode, as described herein.

[0174] The vehicle 500 may further include an instrument cluster 532 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 532 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 532 may include a number of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag system (SRS) information, lighting controls, safety system controls, navigation information, etc. In some examples, information from the infotainment SoC 530 and the instrument cluster 532 may be displayed and / or shared. In other words, the instrument cluster 532 may be included as part of the infotainment SoC 530, or vice versa.

[0175] Fig. 5D is a system diagram for communication between the one or more cloud-based servers and the example autonomous vehicle 500 of Fig. 5A, according to some embodiments of the present disclosure. The system 576 may include the one or more servers 578, the one or more networks 590, and the vehicles, including the vehicle 500. The server(s) 578 may include a plurality of GPUs 584(A)-584(H) (collectively referred to herein as GPUs 584), PCIe switches 582(A)-582(H) (collectively referred to herein as PCIe switches 582), and / or CPUs 580(A)-580(B) (collectively referred to herein as CPUs 580). The GPUs 584, the CPUs 580, and the PCIe switches may be interconnected with high-speed interconnects, such as, without limitation, the NVIDIA-developed NVLink interfaces 588 and / or PCIe interconnects 586. In some examples, the GPUs 584 are connected via NVLink and / or NVSwitch SoC, and the GPUs 584 and the PCIe switches 582 are connected via PCIe links.Although eight GPUs 584, two CPUs 580, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 578 may include any number of GPUs 584, CPUs 580, and / or PCIe switches. For example, the one or more servers 578 may each include eight, sixteen, thirty-two, and / or more GPUs 584.

[0176] The one or more servers 578 may receive, via the one or more networks 590 and from the vehicles, image data representative of images depicting unexpected or changed road conditions, such as recently commenced roadwork. The one or more servers 578 may transmit, via the one or more networks 590 and to the vehicles, neural networks 592, updated neural networks 592, and / or map information 594 containing information about traffic and road conditions. The updates to the map information 594 may include updates to the HD map 522, such as information about construction, potholes, detours, flooding, and / or other obstacles.In some examples, the neural networks 592, the updated neural networks 592, and / or the map information 594 may result from new training and / or experience represented in the data received from any number of vehicles in the area and / or based on training performed in a data center (e.g., using the one or more servers 578 and / or other servers). As explained herein, one or more neural networks 592 and / or one or more updated neural networks 592 may be used to generate priority number values representing probabilities of reuse, costs of eviction, and / or other actions related to storing, deleting, and / or using the map information 594 in the vehicle 500.

[0177] The one or more servers 578 may be used to train machine learning models (e.g., neural networks) based on training data. The training data may be generated by the vehicles and / or in a simulation (e.g., using a game engine). In some examples, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not tagged and / or preprocessed (e.g., if the neural network does not require supervised learning).Training may be performed using one or more classes of machine learning techniques, including, without limitation, supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including replacement dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning models are trained, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles via the one or more networks 590) and / or the machine learning models may be used by the one or more servers 578 to remotely monitor the vehicles.

[0178] In some examples, the one or more servers 578 may receive data from the vehicles and apply the data to real-time, real-time neural networks for intelligent inference. The one or more servers 578 may include deep learning supercomputers and / or dedicated AI computers powered by GPUs 584, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the one or more servers 578 may include a deep learning infrastructure using only CPU-powered data centers.

[0179] The deep learning infrastructure of the one or more servers 578 may be capable of performing rapid inference in real time and may utilize this capability to evaluate and verify the state of the processors, software, and / or associated hardware in the vehicle 500. For example, the deep learning infrastructure may receive periodic updates from the vehicle 500, such as a sequence of images and / or objects that the vehicle 500 has located in that sequence of images (e.g., via computer vision and / or other machine learning object classification techniques).The deep learning infrastructure may run its own neural network to identify the objects and compare them to the objects identified by the vehicle 500, and if the results do not match and the infrastructure concludes that the AI in the vehicle 500 is not functioning properly, the one or more servers 578 may send a signal to the vehicle 500 instructing a fail-safe computer of the vehicle 500 to take control, notify passengers, and perform a safe parking maneuver.

[0180] For inferencing, the one or more servers 578 may include GPUs 584 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-driven servers and inference accelerators may enable real-time responsiveness. In other examples, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors may be used for inferencing. EXAMPLE CALCULATION DEVICE

[0181] Fig. 6 is a block diagram of an example computing device 600 suitable for use in implementing some embodiments of the present disclosure. The computing device 600 may include an interconnect system 602 that directly or indirectly couples the following devices: memory 604, one or more central processing units (CPUs) 606, one or more graphics processing units (GPUs) 608, a communications interface 610, input / output (I / O) ports 612, input / output components 614, a power supply 616, one or more presentation components 618 (e.g., display(s)), and one or more logic units 620. In at least one embodiment, the one or more computing devices 600 may include one or more virtual machines (VMs), and / or each of the components thereof may include virtual components (e.g., virtual hardware components).As non-limiting examples, one or more of the GPUs 608 may include one or more vGPUs, one or more of the CPUs 606 may include one or more vCPUs, and / or one or more of the logic units 620 may include one or more virtual logic units. Thus, a computing device 600 may include discrete components (e.g., a full GPU associated with the computing device 600), virtual components (e.g., a portion of a GPU associated with the computing device 600), or a combination thereof.

[0182] Although the different blocks of Fig. 6 are shown as being connected to wires via the interconnect system 602, this is not intended as a limitation and is for clarity only. For example, in some embodiments, a presentation component 618, such as a display device, may be considered an I / O component 614 (e.g., if the display is a touchscreen). As another example, the CPUs 606 and / or GPUs 608 may include memory (e.g., the memory 604 may represent a storage device in addition to the memory of the GPUs 608, the CPUs 606, and / or other components). In other words, the computing device of Fig. 6 is merely illustrative. It does not distinguish between categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, since all devices within the scope of the computing device are covered by Fig. 6 may be considered.

[0183] The interconnect system 602 may represent one or more connections or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 602 may include one or more bus or connection types, such as an Industry Standard Architecture (ISA) bus, an Extended ISA bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, and / or another type of bus or connection. In some embodiments, there are direct connections between components. For example, the CPU 606 may be directly connected to the memory 604. Further, the CPU 606 may be directly connected to the GPU 608.For a direct or point-to-point connection between components, interconnect system 602 may include a PCIe link to establish the connection. In these examples, a PCI bus need not be included in computing device 600.

[0184] Memory 604 may include a variety of computer-readable media. The computer-readable media may be any available media accessible by the computing device 600. The computer-readable media may include both volatile and non-volatile media, as well as removable and non-removable media. By way of example and without limitation, the computer-readable media may include computer storage media and communication media.

[0185] The computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, the memory 604 may store computer-readable instructions (e.g., representing one or more programs) and / or one or more program elements, such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 600.As used herein, computer storage media do not include signals per se. As explained herein, memory 604 may store map tiles, layers, and / or other map data units for use in performing geolocation, navigation, planning, and / or other tasks on a location-aware system. These map data units may be selectively stored in and / or evicted from memory 604 based on attributes and / or priority number values associated with the map data units.

[0186] The computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and may include any media for conveying information. The term "modulated data signal" may refer to a signal having one or more of its characteristics adjusted or altered to encode information in the signal. The computer storage media may include, for example, and is not limited to, wired media, such as a wired network or a direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of the above should also be within the scope of computer-readable media.

[0187] The one or more CPUs 606 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 600 to perform one or more of the methods and / or processes described herein. The one or more CPUs 606 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of concurrently executing a plurality of software threads. The one or more CPUs 606 may include any type of processor and may include different types of processors depending on the type of computing device 600 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).Depending on the type of computing device 600, the processor may be, for example, an Advanced RISC Machines (ARM) processor implemented with Reduced Instruction Set Computing (RISC) or an x86 processor implemented with Complex Instruction Set Computing (CISC). Computing device 600 may include one or more CPUs 606, in addition to one or more microprocessors or additional coprocessors, such as math coprocessors.

[0188] In addition or alternatively to the one or more CPUs 606, the one or more GPUs 608 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 600 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 608 may be an integrated GPU (e.g., with one or more of the CPUs 606) and / or one or more of the GPUs 608 may be a discrete GPU. In embodiments, one or more of the GPUs 608 may be a co-processor of one or more of the CPUs 606. The one or more GPUs 608 may be used by the computing device 600 to render graphics (e.g., 3D graphics) or perform general-purpose computations. The one or more GPUs 608 may be used, for example, for general-purpose computing on GPUs (GPGPU).The one or more GPUs 608 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The one or more GPUs 608 may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the one or more CPUs 606 received via a host interface). The one or more GPUs 608 may include graphics memory, such as display memory, for storing pixel data or other suitable data, such as GPGPU data. The display memory may be included as part of the memory 604. The one or more GPUs 608 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or connect the GPUs via a switch (e.g., using NVSwitch).When combined, each GPU can generate 608 pixel data or GPGPU data for different sections of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can contain its own memory or share memory with other GPUs.

[0189] In addition to or alternatively to the one or more CPUs 606 and / or the one or more GPUs 608, the one or more logic units 620 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 600 to perform one or more of the methods and / or processes described herein. In embodiments, the one or more CPUs 606, the GPUs 608, and / or the one or more logic units 620 may discretely or jointly execute any combination of the methods, processes, and / or portions thereof. One or more of the logic units 620 may be part of and / or integrated with one or more of the CPUs 606 and / or one or more of the GPUs 608, and / or one or more of the logic units 620 may be discrete components or otherwise external to the CPUs 606 and / or the GPUs 608.In embodiments, one or more of the logic units 620 may be a co-processor of one or more of the CPUs 606 and / or one or more of the GPUs 608.

[0190] Examples of the one or more logic units 620 include one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (Application-Specific Integrated Circuits, ASICs), Floating Point Units (FPUs),Input / output (I / O) elements, peripheral component interconnect (PCI) or PCI Express (PCIe) elements, and / or similar.

[0191] The communication interface 610 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 600 to communicate with other computing devices over an electronic network, including wired and / or wireless communication. The communication interface 610 may include components and functions that enable communication over a variety of networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.In one or more embodiments, the one or more logic units 620 and / or the communication interface 610 may include one or more data processing units (DPUs) to transfer data received over a network and / or via the interconnect system 602 directly to one or more GPUs 608 (e.g., a memory thereof).

[0192] The I / O ports 612 may enable the computing device 600 to be logically coupled to other devices, including the I / O components 614, the one or more presentation components 618, and / or other components, some of which may be built into (e.g., integrated) the computing device 600. Illustrative I / O components 614 include a microphone, a mouse, a keyboard, a joystick, a gamepad, a game controller, a satellite dish, a scanner, a printer, a wireless device, etc. The I / O components 614 may provide a natural user interface (NUI) that processes air gestures, speech, or other physiological inputs generated by a user. In some cases, the inputs may be communicated to a suitable network element for further processing.An NUI may implement any combination of speech capture, stylus capture, facial capture, biometric capture, both on-screen and off-screen gesture capture, air gestures, head and eye tracking, and touch capture (as further described below) associated with a display of the computing device 600. The computing device 600 may include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture capture and recognition. Additionally, the computing device 600 may include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion capture. In some examples, the output of the accelerometers or gyroscopes from the computing device 600 may be used to present immersive augmented reality or virtual reality.

[0193] Power supply 616 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 616 may supply power to computing device 600 to enable operation of the components of computing device 600.

[0194] The one or more presentation components 618 may include a display (e.g., a monitor, a touchscreen, a television monitor, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The one or more presentation components 618 may receive data from other components (e.g., the one or more GPUs 608, the one or more CPUs 606, DPUs, etc.) and output the data (e.g., as an image, video, audio, etc.). EXEMPLARY DATA CENTER

[0195] Fig. Figure 7 illustrates an example data center 700 that may be used in at least one embodiment of the present disclosure. Data center 700 may include a data center infrastructure layer 710, a framework layer 720, a software layer 730, and / or an application layer 740.

[0196] As in Fig. 7, the data center infrastructure layer 710 may include a resource orchestrator 712, clustered computing resources 714, and node computing resources (“node CRs”) 716(1)-716(N), where “N” represents any positive integer. In at least one embodiment, the Node CRs 716(1)-716(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. In some embodiments, one or more Node Cs mayRs among the node CRs 716(1)-716(N) may correspond to a server having one or more of the computing resources mentioned above. Furthermore, in some embodiments, the node CRs 716(1)-716(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or the like, and / or one or more of the node CRs 716(1)-716(N) may correspond to a virtual machine (VM).

[0197] In at least one embodiment, the grouped computing resources 714 may include separate groupings of node CRs 716 housed in one or more racks (not shown) or in many racks in data centers in different geographic locations (also not shown). Separate groupings of node CRs 716 within grouped computing resources 714 may include grouped computing, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs 716, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads.The one or more racks may also contain any number of power modules, cooling modules, and / or network switches in any combination.

[0198] The resource orchestrator 712 may configure or otherwise control one or more node CRs 716(1)-716(N) and / or grouped computing resources 714. In at least one embodiment, the resource orchestrator 712 may include an entity for managing the software design infrastructure (SDI) for the data center 700. The resource orchestrator 712 may include hardware, software, or a combination thereof.

[0199] In at least one embodiment, as in Fig. 7, the framework layer 720 may include a job scheduler 733, a configuration manager 734, a resource manager 736, and / or a distributed file system 738. The framework layer 720 may include a framework that supports the software 732 of the software layer 730 and / or one or more applications 742 of the application layer 740. The software 732 or the one or more applications 742 may each include web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 720 may be some type of free and open source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter "Spark"), which may utilize a distributed file system 738 for processing large amounts of data (e.g., "Big Data").In at least one embodiment, the job scheduler 733 may include a Spark driver to facilitate the scheduling of workloads supported by various layers of the data center 700. The configuration manager 734 may be capable of configuring various layers, such as the software layer 730 and the framework layer 720, which includes Spark and the distributed file system 738, to support the processing of large amounts of data. The resource manager 736 may be capable of managing clustered or grouped computing resources allocated or assigned to support the distributed file system 738 and the job scheduler 733. In at least one embodiment, the clustered or grouped computing resources may include the clustered computing resource 714 at the infrastructure layer 710 of the data center.The resource manager 736 may coordinate with the resource orchestrator 712 to manage these allocated or assigned computing resources.

[0200] In at least one embodiment, the software 732 included in software layer 730 may include software used by at least portions of node CRs 716(1)-716(N), clustered computing resources 714, and / or distributed file system 738 of framework layer 720. One or more types of software may include, among others, web page searching software, email virus scanning software, database software, and streaming video content software.

[0201] In at least one embodiment, the applications 742 included in the application layer 740 may include one or more types of applications used by at least portions of the node CRs 716(1)-716(N), the clustered computing resources 714, and / or the distributed file system 738 of the framework layer 720. One or more types of applications may include, but are not limited to, any number of genomic applications, cognitive computation, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in connection with one or more embodiments.

[0202] In at least one embodiment, one of a configuration manager 734, resource manager 736, and resource orchestrator 712 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically feasible manner. Self-modifying actions may relieve a data center operator of data center 700 from making potentially poor configuration decisions and potentially avoid underutilized and / or poorly performing sections of a data center.

[0203] Data center 700 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, one or more machine learning models may be trained by calculating weighting parameters according to a neural network architecture using software and / or computing resources described above with reference to data center 700.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with reference to data center 700 using weighting parameters calculated by one or more training techniques, such as, but not limited to, those described herein. For example, the trained or deployed machine learning models may be used to generate priority number values that can be used to manage the storage and eviction of map data units in memory on one or more remote location-aware systems.

[0204] In at least one embodiment, data center 700 may utilize CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inferencing using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as a service to enable users to train or infer information, such as image capture, speech capture, or other artificial intelligence services. EXAMPLE NETWORK ENVIRONMENTS

[0205] Network environments suitable for implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of the one or more computing devices 600 of Fig. 6 - for example, each device may include similar components, features, and / or functionality of the one or more computing devices 600. Furthermore, if backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be included as part of a data center 700, an example of which is described herein with reference to Fig. 7 is described in more detail.

[0206] The components of a network environment can communicate with each other over one or more networks, which can be wired, wireless, or both. The network can contain multiple networks or a network of networks. For example, the network can contain one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or a public switched telephone network (PSTN), and / or one or more private networks. If the network contains a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0207] Compatible network environments may include one or more peer-to-peer network environments—in which case, a server cannot be included in a network environment—and one or more client-server network environments—in which case, one or more servers can be included in a network environment. In peer-to-peer network environments, the functionality described herein with respect to one or more servers may be implemented on any number of client devices.

[0208] In at least one embodiment, a network environment may include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. A framework layer may include a framework for supporting software of a software layer and / or one or more applications of an application layer. The software or the one or more applications may each include web-based service software or applications. In embodiments, one or more of the client devices may utilize the web-based service software or applications (e.g.,by accessing the service software and / or applications through one or more application programming interfaces (APIs). The framework layer may be some type of free and open-source software web application framework, e.g., using, but not limited to, a distributed file system for processing large amounts of data (e.g., "Big Data").

[0209] A cloud-based network environment may provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Each of these various functions may be distributed across multiple locations of central or core servers (e.g., one or more data centers that may be located across a state, region, country, globe, etc.). When a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers may offload at least a portion of the functionality to the one or more edge servers. A cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0210] The one or more client devices may include at least some of the components, features, and functions of the one or more described herein with respect to Fig.6. By way of example, and not limitation, a client device may be embodied as a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or global positioning device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, a hydrofoil, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or gaming system, an entertainment system, a vehicle computing system, an embedded system controller, a remote control, an appliance, a consumer electronics device, a workstation, an edge device,any combination of these described devices or any other suitable device.

[0211] In summary, the disclosed techniques calculate priority scores for discrete units of map data downloaded in response to vehicle movement and stored in the memory of an in-vehicle SoC (or other location-aware system). These map data units may include map tiles, layers within map tiles, and / or other discrete pieces of map data. The priority scores may be calculated based on attributes of the map data units, such as (but not limited to) recency of use, frequency of use, distance from a current location of the location-aware system, overlap with a frequent or known route of the location-aware system, a map data unit size, a time of day, and / or a day of the week.Each priority number value may represent a probability of use for a corresponding map data unit and / or another measure of a potential "cost" associated with clearing the map data unit. One or more map data units with priority number values representing the lowest potential cost or cost for clearing may then be deleted in response to new map data being stored in memory.

[0212] A technical advantage of the disclosed techniques relative to prior approaches is the ability to effectively utilize limited memory for storing and updating map data in a location-aware system. In this regard, the techniques prioritize map data units for eviction based on various attributes relevant to the subsequent use of the map data units and / or the cost associated with downloading the map data units, thereby reducing the likelihood that evicted map data must be subsequently re-downloaded. Consequently, the disclosed techniques improve data reuse, network utilization, and resource overhead compared to conventional, naive approaches for managing and updating stored map data.

[0213] The disclosure of the present application also includes the following numbered clauses:

[0214] Clause 1.In some embodiments, a method comprises determining a corresponding set of attributes for individual map data units of a plurality of map data units stored in memory of a location-aware system; for the individual map data units included in the plurality of map data units, calculating a priority number value for the individual map data unit based on the set of attributes corresponding to the individual map data unit; determining, based on the plurality of priority number values for the plurality of map data units, one or more map data units to be evicted from memory; and causing at least a portion of the one or more map data units to be deleted from memory or replaced in memory in response to receiving one or more new map data units for storage in memory.

[0215] Clause 2. The method of clause 1, wherein calculating the priority number value comprises inputting the corresponding set of attributes for the individual map data unit into a machine learning model and generating the priority number value representing a likelihood of reusing the individual map data unit by executing the machine learning model.

[0216] Clause 3. The method of any one of clauses 1 to 2, further comprising determining a plurality of statistics associated with use of a second plurality of map data units by one or more location-aware systems; and updating one or more parameters of a machine learning model based at least on the plurality of statistics and the plurality of sets of attributes, wherein the priority number value is calculated using the updated machine learning model based on at least one input including the corresponding set of attributes.

[0217] Clause 4. The method of any of clauses 1 to 3, wherein the priority number value is calculated based on at least one weighted combination of the corresponding set of attributes.

[0218] Clause 5. The method of any one of clauses 1 to 4, wherein determining the one or more map data units to be evicted comprises determining one or more priority number values from the plurality of priority number values indicative of a lowest cost value associated with evicting the one or more map data units.

[0219] Clause 6. The method of any one of clauses 1 to 5, wherein determining the one or more map data units to be cleared comprises determining one or more layers corresponding to the one or more map data units to be cleared from a map tile based on a set of additional attributes associated with the one or more layers.

[0220] Clause 7. A method according to any one of clauses 1 to 6, wherein the set of additional attributes comprises at least one of a layer size, a layer importance, a number of requests associated with a layer, an aging of a layer, or a download time associated with a layer.

[0221] Clause 8. The method of any one of clauses 1 to 7, wherein the plurality of map data units comprises at least one of a plurality of map tiles or a plurality of layers included in the plurality of map tiles.

[0222] Clause 9. The method of any one of clauses 1 to 8, wherein each set of attributes included in the plurality of sets of attributes comprises at least one of a last time of use, a frequency of use, a geographical distance from a current location of the location-aware system, an overlap with a route associated with a vehicle, a map data unit size, a time of day, or a day of the week.

[0223] Clause 10. A method according to any one of clauses 1 to 9, wherein the location-aware system comprises at least one of a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a mobile device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot;a system for performing operations with conversational AI; a system for generating synthetic data; a system that includes one or more virtual machines (VMs); a system that is implemented at least in part in a data center; or a system that is implemented at least in part using cloud computing resources.

[0224] Clause 11. In some embodiments, a processor comprises one or more circuits for performing operations comprising determining a corresponding set of attributes for each of a plurality of map data units stored in memory of a location-aware system; for each map data unit included in the plurality of map data units, calculating a priority number value for the map data unit based on the set of attributes corresponding to the map data unit; determining, based on the plurality of priority number values for the plurality of map data units, one or more map data units to be evicted from memory;and causing at least a portion of the one or more map data units to be deleted from memory or replaced in memory in response to receiving one or more new map data units for storage in memory;

[0225] Clause 12. The processor of Clause 11, wherein calculating the priority number value comprises inputting the set of attributes corresponding to the map data unit into a machine learning model and generating the priority number value representing a probability of reusing the map data unit by executing the machine learning model.

[0226] Clause 13. The processor of any of clauses 11 to 12, wherein the operations further comprise determining a plurality of statistics associated with use of a second plurality of map data units by one or more location-aware systems; and updating one or more parameters of a machine learning model based on the plurality of statistics and the plurality of sets of attributes, wherein the priority number value is calculated by the updated machine learning model based on at least one input including the corresponding set of attributes.

[0227] Clause 14. The processor of any of clauses 11 to 13, wherein determining the one or more map data units to be evicted comprises determining, based on at least a subset of the plurality of priority number values for a plurality of map tiles included in the plurality of map data units, a priority number value indicative of a lowest cost associated with evicting a corresponding map tile included in the plurality of map tiles; and determining, based on at least a plurality of layers included in the corresponding map tile, a layer to be evicted from memory.

[0228] Clause 15. The processor of any of clauses 11 to 14, wherein the tier to be evicted from memory is determined based on at least a second subset of the plurality of priority number values for the plurality of tiers.

[0229] Clause 16. The processor of any of clauses 11 to 15, wherein the set of attributes corresponding to each of the plurality of tiers comprises at least one of a tier size, a tier importance, a number of requests associated with a tier, an aging of a tier, or a download time associated with a tier.

[0230] Clause 17. The processor of any one of clauses 11 to 16, wherein the set of attributes corresponding to each of the plurality of map tiles comprises at least one of a last time of use, a frequency of use, a geographical distance from a current location of the location-aware system, an overlap with a route associated with a vehicle, a map tile size, a time of day, or a day of week.

[0231] Clause 18. A processor according to any one of clauses 11 to 17, wherein the processor comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more conversational AI operations;a system for performing one or more operations with generative AI; a system implementing one or more large language models (LLMs); a system for generating synthetic data; a system including one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources. Clause 19. In some embodiments, a system comprises one or more processing units for performing operations comprising determining a corresponding set of attributes for each of a plurality of map data units stored in memory in a location-aware system;for each map data unit included in the plurality of map data units, calculating a priority number value for the map data unit based on the set of attributes corresponding to the map data unit; determining, based on the plurality of priority number values for the plurality of map data units, one or more map data units to be evicted from memory; and causing at least a portion of the one or more map data units to be deleted from memory or replaced in memory in response to receiving one or more new map data units for storage in memory;

[0232] Clause 20. A system according to Clause 19, wherein the system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more conversational AI operations;a system for performing one or more generative AI operations; a system implementing one or more large language models (LLMs); a system for generating synthetic data; a system including one or more virtual machines (VMs); a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

[0233] The disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., and refer to code that performs specific tasks or implements specific abstract data types. The disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc.The disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices that are interconnected via a network for communication.

[0234] As used herein, any reference to "and / or" in reference to two or more elements should be interpreted to mean only one element or a combination of elements. For example, "Element A, Element B, and / or Element C" may include only Element A, only Element B, only Element C, Element A and Element B, Element A and Element C, Element B and Element C, or Elements A, B, and C. Furthermore, "at least one of Element A or Element B" may include at least one of Element A, at least one of Element B, or at least one of Element A and at least one of Element B. Further, "at least one of Element A and Element B" may include at least one of Element A, at least one of Element B, or at least one of Element A and at least one of Element B.

[0235] The subject matter of the present disclosure is specifically described herein to satisfy legal requirements. However, the description itself is not intended to limit the scope of the present disclosure. Rather, the inventors have contemplated that the claimed subject matter may be embodied in other ways to include various steps or combinations of steps similar to those described herein, in conjunction with other present or future technologies. Although the terms "step" and / or "block" may be used herein to refer to various elements of the methods employed, the terms should not be interpreted to imply any particular ordering among or between the various steps disclosed herein, unless and except that the order of the individual steps is expressly described.

[0236] It is to be understood that aspects and embodiments described above are purely exemplary and that modifications of details may be made within the scope of the claims.

[0237] Each device, method, and feature disclosed in the description, and (where appropriate) the claims and drawings may be provided independently or in any suitable combination.

[0238] Reference signs appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 16 / 101,232

[0116] Cited non-patent literature

[0000] Society of Automotive Engineers, SAE) (Standard No. J3016-201806, published on June 15, 2018, Standard No. 13016-201609, published on September 30, 2016

[0074]

Claims

[1] Method comprising: determining a corresponding set of attributes for individual map data units of a plurality of map data units stored in memory of a location-aware system; for the individual map data units included in the plurality of map data units, calculating a priority number value for the individual map data unit based on the set of attributes corresponding to the individual map data unit; Determining, based on the plurality of priority number values for the plurality of map data units, one or more map data units to be flushed from the memory; and Causing at least a portion of the one or more map data units to be deleted from memory or replaced in memory in response to receiving one or more new map data units for storage in memory [2] The method of claim 1, wherein calculating the priority number value comprises: Entering the corresponding set of attributes for the individual map data unit into a machine learning model and Generating the priority number value representing a probability of reusing the individual map data unit by executing the machine learning model. [3] Method according to one of the preceding claims, further comprising: Determining a plurality of statistics associated with the use of a second plurality of map data units by one or more location-aware systems; and Updating one or more parameters of a machine learning model based on at least the plurality of statistics and the plurality of sets of attributes, wherein the priority number value is calculated using the updated machine learning model based on at least one input containing the corresponding set of attributes [4] A method according to any one of the preceding claims, wherein the priority number value is calculated based on at least a weighted combination of the corresponding set of attributes. [5] The method of any preceding claim, wherein determining the one or more map data units to be cleared comprises determining one or more priority number values from the plurality of priority number values indicating a lowest cost value associated with clearing the one or more map data units. [6] The method of any preceding claim, wherein determining the one or more map data units to be cleared comprises determining one or more layers corresponding to the one or more map data units to be cleared from a map tile based on a set of additional attributes associated with the one or more layers. [7] The method of claim 6, wherein the set of additional attributes comprises at least one of a layer size, a layer importance, a number of requests associated with a layer, an aging of a layer, or a download time associated with a layer. [8] A method according to any one of the preceding claims, wherein the plurality of map data units comprises at least one of a plurality of map tiles or a plurality of layers included in the plurality of map tiles. [9] A method according to any one of the preceding claims, wherein each set of attributes included in the plurality of sets of attributes comprises at least one of a last time of use, a frequency of use, a geographical distance from a current location of the location-aware system, an overlap with a route associated with a vehicle, a map data unit size, a time of day, or a day of the week. [10] A method according to any one of the preceding claims, wherein the location-aware system comprises at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a mobile device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing operations using conversational AI; a system for generating synthetic data; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources. [11] Processor comprising: one or more circuits for performing operations, comprising: determining a corresponding set of attributes for each of a plurality of map data units stored in memory of a location-aware system; for each map data unit included in the plurality of map data units, calculating a priority number value for the map data unit based on the set of attributes corresponding to the map data unit; Determining, based on the plurality of priority number values for the plurality of map data units, one or more map data units to be flushed from the memory; and Causing at least a portion of the one or more map data units to be deleted from memory or replaced in memory in response to receiving one or more new map data units for storage in memory [12] The processor of claim 11, wherein calculating the priority number value comprises: Inputting the set of attributes corresponding to the map data unit into a machine learning model and Generating the priority number value representing a probability of reusing the map data unit by executing the machine learning model. [13] A processor according to any one of claims 11 or 12, wherein the operations further comprise: Determining a plurality of statistics associated with the use of a second plurality of map data units by one or more location-aware systems; and Updating one or more parameters of a machine learning model based on the plurality of statistics and the plurality of sets of attributes, wherein the priority number value is calculated by the updated machine learning model based on at least one input including the corresponding set of attributes. [14] A processor according to any one of claims 11 to 13, wherein determining the one or more map data units to be cleared comprises: Determining, based on at least a subset of the plurality of priority number values for a plurality of map tiles included in the plurality of map data units, a priority number value indicative of a lowest cost associated with clearing a corresponding map tile included in the plurality of map tiles; and Determining, based on at least a plurality of layers contained in the corresponding map tile, a layer to be evicted from memory. [15] The processor of claim 14, wherein the layer to be evicted from memory is determined based on at least a second subset of the plurality of priority number values for the plurality of layers. [16] The processor of claim 15, wherein the set of attributes corresponding to each of the plurality of layers comprises at least one of a layer size, a layer importance, a number of requests associated with a layer, an aging of a layer, or a download time associated with a layer. [17] The processor of any of claims 14 to 16, wherein the set of attributes corresponding to each of the plurality of map tiles comprises at least one of a last time of use, a frequency of use, a geographic distance from a current location of the location-aware system, an overlap with a route associated with a vehicle, a map tile size, a time of day, or a day of the week. [18] A processor according to any one of claims 11 to 17, wherein the processor comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more operations using conversational AI; a system for performing one or more operations using generative AI;*** a system that implements one or more large-scale language models (LLMs); a system for generating synthetic data; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources. [19] System comprising: one or more processing units for performing operations, comprising: determining a corresponding set of attributes for each of a plurality of map data units stored in a memory in a location-aware system; for each map data unit included in the plurality of map data units, calculating a priority number value for the map data unit based on the set of attributes corresponding to the map data unit; Determining, based on the plurality of priority number values for the plurality of map data units, one or more map data units to be flushed from the memory; and Causing at least a portion of the one or more map data units to be deleted from memory or replaced in memory in response to receiving one or more new map data units for storage in memory [20] A system according to any one of claims 19, wherein the system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system for generating or presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system implemented using a robot; a system for performing one or more operations using conversational AI; a system for performing one or more operations using generative AI; a system that implements one or more large language models (LLMs); a system for generating synthetic data; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources.

Images