DETERMINING WAIT SITUATION INFORMATION ASSOCIATED WITH TRAFFIC CHARACTERISTICS FOR AUTONOMOUS SYSTEMS AND APPLICATIONS

Abstract

In various examples, determining wait situation information associated with traffic features for autonomous and semi-autonomous systems and applications is described herein. The systems and methods described herein may process data representing actual driving behaviors associated with machine users to determine wait situation information, such as wait lines (e.g., stop lines, etc.), for traffic features located within an environment. For example, map streams (e.g., trips, etc.) associated with machines navigating near a traffic feature may be evaluated based at least on whether the rules associated with the environment and / or the traffic feature were followed. At least a portion of the map streams, such as map streams associated with at least one threshold number value, may then be used to determine a wait line associated with the traffic feature.In addition, map data representing a map can be updated to indicate the location of the waiting line within the surrounding area.

Description

BACKGROUND

[0001] In order for an autonomous or semi-autonomous vehicle to safely navigate through an environment, the vehicle may refer to maps, such as navigation maps, standard definition (SD), and / or high definition (HD) maps, that correspond to the environment in which the vehicle intends to operate. Due to the detailed, three-dimensional, and highly accurate nature of an HD map, navigating according to the HD map has been proven effective for safe navigation in environments where HD map information is available. For example, the vehicle may refer to the map to determine the locations of traffic features such as lanes, intersections, traffic signals, traffic signs, lane lines, and / or the like located within an environment. The vehicle may then use the locations of these traffic features when determining how to navigate within the environment.For example, when approaching an intersection that contains a traffic signal, the vehicle may use the locations of the intersection, lane, and / or traffic signal to determine how to navigate through the intersection.

[0002] Under certain circumstances, an HD map may contain additional information associated with an environment, such as areas within the environment where vehicles may be required to yield and / or stop. These areas may be regulated, for example, by traffic signals, traffic signs, yield signs, crosswalks, and / or other types of traffic control features. Therefore, vehicles may further be required to follow rules associated with the various features when approaching these areas of the environment. For example, when a vehicle is approaching an intersection regulated by a traffic signal and / or a stop sign, it may be important for the vehicle to determine a stop location.However, based on the design of the immediate area surrounding the intersection, it may be difficult for the vehicle to determine the appropriate stopping location, which may reduce vehicle and / or pedestrian safety and / or cause problems navigating the area. SUMMARY

[0003] The invention is defined by the claims. To illustrate the invention, aspects and embodiments are described herein, which may or may not be within the scope of the claims.

[0004] The embodiments of the present disclosure relate to determining wait situation information associated with traffic features for autonomous and semi-autonomous systems and applications. The systems and methods described herein may process data associated with actual driving behaviors of drivers of machines to determine wait situation information, such as wait lines (e.g., yield lines, stop lines, etc.) associated with traffic features located within an environment. For example, map streams (e.g., sensor data, perception results, etc., corresponding to trips, etc.) associated with machines navigating near a traffic feature, such as a traffic sign and / or a traffic light, may be evaluated based at least on whether the rules associated with the environment and / or traffic feature have been followed. At least a portion of the map streams, such asMapstreams associated with at least one threshold value can then be used to determine a wait line associated with the traffic feature. In some examples, the wait line may include a physical line within the environment (e.g., a traffic line), while in other examples, the wait line may include a projected line within the environment. This process can then be repeated to determine any number of wait lines for any number of traffic features. Furthermore, map data representing a map can be updated to further indicate the locations of the wait lines within the environment.

[0005] Unlike conventional systems, the systems of the current disclosure, in some embodiments, are capable of automatically determining the locations of wait lines for traffic features associated with wait situations and / or are capable of updating maps to indicate the locations of the wait lines. Therefore, vehicles using the maps of the current disclosure are better able to determine how to navigate through areas of the environments associated with these traffic features, such as by determining appropriate locations to yield and / or stop when required by the rules of the environments and / or the traffic features. For example, and in conventional systems, upon approaching a traffic feature, such as when the traffic feature is a stop sign and / or a traffic light that is red, a vehicle may simply stop at a random location.In contrast, using the maps of the current disclosure, the vehicle may determine the appropriate location to stop, which may increase safety for users and / or the vehicle and / or improve the way the vehicle navigates within the environment and with respect to the traffic feature.

[0006] The disclosure extends to all novel aspects or features described and / or illustrated herein.

[0007] Further features of the disclosure are characterized by the independent and dependent claims.

[0008] Any feature of one aspect of the disclosure may be applied to other aspects of the disclosure in any suitable combination. In particular, method aspects may be applied to device or system aspects, and vice versa.

[0009] Furthermore, features implemented in hardware may be implemented in software, and vice versa. Any reference to software and hardware features herein should be construed accordingly.

[0010] Any system or device feature described herein may also be provided as a method feature, and vice versa. System and / or device aspects described functionally (including means plus functional features) may alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0011] It is also to be understood that certain combinations of the various features described and defined in each aspect of the disclosure may be implemented and / or provided and / or used independently of one another.

[0012] The disclosure also provides computer programs and computer program products comprising software code configured to, when executed on a data processing device, perform any of the methods described herein and / or embody any of the device and system features described herein, including any or all of the component steps of a method.

[0013] The disclosure also includes a computer or computer system (including networked or distributed systems) having an operating system that supports a computer program for performing the methods described herein and / or for embodying the device or system features described herein.

[0014] The disclosure also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0015] The disclosure also provides a signal carrying one or more of the aforementioned computer programs.

[0016] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0017] Aspects and embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present systems and methods for determining waiting situation information associated with traffic characteristics for autonomous and semi-autonomous systems and applications are described in detail below with reference to the attached drawings. They show: Fig. 1 is an exemplary data flow diagram for a process for determining wait situation information associated with traffic features located within an environment, according to some embodiments of the present disclosure; Fig. 2 illustrates an example of a map associated with an environment including waiting situations, in accordance with some embodiments of the present disclosure; Fig. 3 illustrates an example of associating trips with traffic features, according to some embodiments of the present disclosure; Fig. 4 illustrates an example of grouping paths associated with a traffic feature located within an environment, according to some embodiments of the present disclosure; Fig. 5 illustrates an example of determining candidate lines associated with traffic features located within an environment, according to some embodiments of the present disclosure; Fig. 6 illustrates an example of using paths to determine a location for projecting a wait line within an environment, according to some embodiments of the present disclosure; Fig. 7 illustrates an example of a map updated to indicate wait lines associated with traffic features, in accordance with some embodiments of the present disclosure; Fig. 8 is a flowchart illustrating a method for determining wait situation information associated with a traffic feature, according to some embodiments of the present disclosure; Fig. 9 is a flowchart illustrating a method for determining a wait line for a traffic feature associated with a wait situation, according to some embodiments of the present disclosure; Fig. 10A is an illustration of an exemplary autonomous vehicle, according to some embodiments of the present disclosure; Fig. 10B shows an example of camera locations and fields of view for the exemplary autonomous vehicle from Fig. 10A, according to some embodiments of the present disclosure; Fig. 10C is a block diagram of an exemplary system architecture for the exemplary autonomous vehicle of Fig. 10A, according to some embodiments of the present disclosure; Fig. 10D is a system diagram for communication between one or more cloud-based servers and the exemplary autonomous vehicle of Fig. 10A, according to some embodiments of the present disclosure; Fig. 11 is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure; and Fig. 12 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] Systems and methods are disclosed relating to determining waiting situation information associated with traffic characteristics for autonomous and semi-autonomous systems and applications. Although the present disclosure is made with respect to an exemplary autonomous or semi-autonomous vehicle or machine 1000 (alternatively referred to herein as "vehicle 1000," "ego-vehicle 1000," "ego-machine 1000," or "machine 1000," an example of which is provided with respect to Fig. 10A-10D), this is not limiting. For example, without limitation, the systems and methods described herein may be used by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, guided and unguided robots or robotic platforms, warehouse vehicles, off-highway vehicles, vehicles coupled to one or more trailers, hydrofoils, boats, shuttle vehicles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Furthermore, although the present disclosure may be described in terms of generating and / or updating maps, this is not to be construed as limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology areas where object recognition and / or map generation may be used.

[0020] For example, one or more systems may generate, receive, retrieve, and / or maintain map data representing a map (e.g., a navigation map, a standard definition (SD) map, a high definition (HD) map, etc.) of an environment. As described herein, the map may indicate information such as poses (e.g., locations, orientations, etc.) and / or dimensions associated with objects located within the environment. For example, the map may indicate poses associated with traffic features such as, but without limitation, traffic signals (e.g., traffic lights), traffic signs, roads, lanes, traffic lines (e.g., lane lines, crosswalk lines, railroad crossing lines, stop lines, etc.), intersections, and / or any other type of traffic features located within the environment.Furthermore, in some examples, the map data may represent associations between the objects located within the environment. For example, the map data may represent an association between a traffic feature and a lane, such as by associating a traffic light, a traffic sign, and / or a traffic line with the lane. Furthermore, in some examples, the map data (and / or other data) may represent rules associated with the environment and / or the objects. For example, the map data may represent directions of travel associated with roads and / or lanes, the course of action when approaching a traffic feature (e.g., stop at a red light, yield at a yellow light, proceed at a green light, proceed in a specified direction, etc.), speed limits, stop durations, and / or any other type of rule.

[0021] The system(s) may further receive data (referred to in some examples as "mapstreams" or "trips") generated using one or more machines (e.g., one or more vehicles) while navigating within the environment. As described herein, a mapstream may include, without limitation, a stream of sensor data (e.g., imagery, LiDAR data, RADAR data, etc.), perception outputs from one or more neural networks, trajectory outputs indicating paths traveled within the environment (e.g., locations, orientations, etc.), velocity information (e.g., speeds, accelerations, etc.), and / or any other data generated by a machine during a trip. In some examples, at least a portion of the mapstreams may be associated with actual human drivers navigating within the environment.In some examples, at least a portion of the mapstreams may be associated with autonomous machines that navigate within the environment.

[0022] The one or more systems may then use the map data and / or the map streams to determine wait situation information associated with the environment. For example, for a traffic feature, such as a traffic light, a traffic sign, and / or any other type of traffic feature that may require a machine to yield and / or stop within the environment, the one or more systems may identify one or more map streams associated with the traffic feature. As described herein, a map stream may be associated with the traffic feature based at least on (1) a machine associated with the map stream navigating in a lane associated with the traffic feature, (2) the machine needing to navigate according to the one or more rules associated with the traffic feature, and / or (3) another technique being used.For example, if the one or more systems use the map data to determine that the traffic feature is associated with a portion of a travel lane within the environment, and the map stream indicates that the machine navigated along the portion of the travel lane, the system may associate the map stream with the traffic feature. In some examples, the one or more systems may also trim the one or more map streams, such as by removing portions of the one or more map streams that are not associated with the traffic feature (e.g., portions of the one or more map streams that are outside a threshold distance from the traffic feature), where the trimmed one or more map streams may be referred to as one or more trips and / or one or more paths.

[0023] The one or more systems may then determine one or more numerical values ​​associated with the one or more trips based at least on the one or more rules associated with the traffic feature and / or the environment. For example, the one or more systems may determine a higher numerical value for a trip in which the one or more rules of the environment and / or traffic feature were followed and a lower numerical value for a trip in which the one or more rules of the environment and / or traffic feature were not followed. For a first example, if the traffic feature includes a stop sign, then the one or more systems may determine a higher numerical value for a trip if a machine decelerated to a stop before the stop sign, waited for a period of time, and then accelerated after the stop sign.For a second example, if the traffic feature again includes a stop sign, then the one or more systems may determine a lower numerical value for a trip if a machine does not completely stop while navigating past the stop sign. In some examples, numerical values ​​may be associated with a range, such as between 0 and 1 (and / or a different range). For example, a high numerical value may be close to a high portion of the range, such as 1, while a lower numerical value may be close to a low portion of the range, such as 0.

[0024] The one or more systems may then use the one or more trips to determine one or more paths associated with the traffic feature and / or a lane associated with the traffic feature. For example, the one or more systems may determine that the one or more paths include at least a portion of the one or more trips associated with the same lane as the traffic feature. The one or more systems may then group and / or merge the one or more paths to generate a final path and / or determine a final numeric value associated with the one or more paths. For example, the one or more systems may determine the final numeric value as the average, the median, the mode, and / or using another measure associated with the one or more numeric values.In some examples, the one or more systems may remove one or more of the one or more paths when performing the grouping. For example, the one or more systems may remove one or more of the one or more paths associated with one or more numerical values ​​that do not meet (e.g., are less than) a threshold numerical value. In some examples, the one or more systems may remove those one or more paths because the one or more paths do not adequately represent how machines should navigate with respect to the traffic feature. The one or more systems may then use the one or more remaining paths and / or the final path generated using the one or more remaining paths to determine wait situation information associated with the traffic feature.

[0025] For example, in some examples, the one or more systems may determine one or more candidate lines that may include a hold line associated with the traffic feature using the map data. As described herein, a candidate line may include a traffic line that at least partially crosses the lane associated with the traffic feature. The one or more systems may then select a candidate line from the one or more candidate lines to include as the hold line based on at least the one or more paths and / or the end path. For a first example, the one or more systems may select the candidate line where a majority of the one or more paths include at least one short stop upon approaching the traffic feature as the hold line.For a second example, the one or more systems may select the candidate line where the final path includes at least one short stop when approaching the traffic feature as the waiting line.

[0026] Additionally or alternatively, in some examples, such as when the map data does not indicate one or more candidate lines and / or the environment does not contain any traffic lines that could serve as the wait line for the traffic feature, the one or more systems may then use one or more additional techniques to determine a location of the wait line within the environment. For a first example, the one or more systems may project a wait line within the environment at a location for which the majority of the one or more paths include at least one short stop upon approaching the traffic feature. For a second example, the one or more systems may project a wait line within the environment at a location for which the final path includes at least one short stop upon approaching the traffic feature.In other words, in each of these examples, the one or more systems may use the actual driving behaviors associated with one or more drivers within the environment to determine the location of the waiting line.

[0027] In some examples, the one or more systems may perform similar processes to determine wait information (e.g., wait lines) associated with additional traffic features located within the environment. Furthermore, in some examples, the one or more systems may combine two or more of these end paths to determine wait information for multiple traffic features located within the environment, such as in a specific region of the environment. For example, the one or more systems may group end paths associated with the same intersection to determine wait information for the entire intersection.In such an example, the wait situation information may include wait lines for different traffic characteristics of the intersection, different roads through the intersection, different lanes through the intersection, and / or so on.

[0028] As described herein, the one or more systems may then update the map data to indicate the wait situation information associated with the traffic feature. In some examples, the one or more systems may update the map data by encoding the map data with the wait situation information. Additionally or alternatively, in some examples, the one or more systems may update the map data by generating and / or updating a layer of the map data associated with wait situations. The one or more systems may then provide the map data updated with the wait situation information to one or more machines navigating within the environment.As described herein, since the map data now indicates the wait situation information, machines navigating within the environment can use the wait situation information to determine locations to stop when approaching traffic features. For a first example, if a machine is approaching a traffic light that is red, the machine may determine to stop at the wait line associated with the traffic light. For a second example, if a machine is approaching a stop sign, the machine may determine to stop at the wait line associated with the stop sign.

[0029] The systems and methods described herein may be used, without limitation, by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), autonomous vehicles or machines, guided and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, hydrofoils, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater vehicles, drones, and / or other types of vehicles.Furthermore, the systems and methods described herein may be used for a variety of purposes, including, but not limited to, machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, light transport simulations (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or other suitable applications.

[0030] The disclosed embodiments may include a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented with a robot, aviation systems, media systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems implementing large language models (LLMs), systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations,Systems for performing light transport simulations, systems for performing collaborative content creation for 3D assets, systems for performing generative AI operations, systems implemented at least partially using cloud computing resources, and / or other types of systems.

[0031] With reference to Fig. 1 illustrates Fig. 1 illustrates an exemplary dataflow diagram for a process 100 for determining wait situation information associated with traffic features located within an environment, according to some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are set forth as examples only. Other arrangements and elements (e.g., engines, interfaces, functions, arrangements, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and location.Various functions described herein that are performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may be performed using similar components, features, and / or functionality as those of the example autonomous vehicle 1000 of FIG. Fig. 10A-10D, the exemplary computing device 1100 of Fig. 11 and / or the exemplary data center 1200 from Fig. 12.

[0032] The process 100 may include a wait situation component 102 receiving map data 104 representing a map associated with an environment. As shown, the map data 104 may include and / or be associated with at least one of map streams 106, object data 108, and wait situation data 110. As described herein, a map stream 106 may include, without limitation, a stream of sensor data (e.g., image data, LiDAR data, RADAR data, motion data, orientation data, location data, etc.), perception outputs from one or more neural networks, trajectory outputs indicating paths traveled within the environment (e.g., locations, orientations, etc.), velocity information (e.g., speeds, accelerations, etc.), and / or any other data generated by a machine during a trip.Furthermore, the object data 108 (which, in some examples, may include a portion of the map streams 106) may represent information associated with objects located within the environment, such as traffic features (e.g., roads, road lines, traffic signs, traffic lights, waiting situations, parking spaces, etc.). As described herein, the information may include, but is not limited to, classifications associated with the objects, poses associated with the object, and / or any other type of information that may be included in a map. For example, a classification may include a road, a road line, a traffic sign, a traffic light, a parking space, a sidewalk, a curb, and / or any other type of object classification. Furthermore, a pose associated with an object may include a location (e.g.,the x-coordinate location, the y-coordinate location and / or the z-coordinate location) and / or an orientation (e.g., roll, pitch, yaw) associated with an object.

[0033] In some examples, the object data 108 may further represent associations between objects. For a first example, and for an intersection, the object data 108 may represent an association between a first traffic light and a first lane, a second traffic light and a second lane, a third traffic light and a third lane, and / or so on. In such an example, the associations may indicate that machines are assumed to navigate according to the traffic lights associated with the lanes for which the machines are navigating. For a second example, and for a traffic line (e.g., a crosswalk line, etc.), the object data 108 may represent an association between the traffic line and a lane of a road within an environment.

[0034] The wait situation data 110 may represent wait situation information associated with wait situations located within the environment. As described herein, a wait situation may include a location of a traffic signal, yield sign, stop sign, construction, crosswalk, railroad crossing, and / or another type of wait situation that may require a machine to yield and / or stop within the environment, or a location of an intersection regulated by them. Furthermore, as described herein, the wait situation component 102 may use the map data 104 to determine additional information associated with the wait situations.For example, the wait situation component 102 may use at least the map streams 106 representing how machines have navigated within the environment to determine at least locations and / or lines for yielding right-of-way and / or stopping with respect to the traffic features associated with the wait situations, where these locations and / or lines may be referred to as "wait locations" and / or "wait lines." In some examples, the wait situation component 102 may use the map streams 106 to better determine the actual locations where machines are expected to stop when approaching the wait situations.

[0035] For example, Fig. 2 illustrates an example of a map 202 associated with an environment including wait situations 204(1)-(2) (also referred to in the singular as "wait situation 204" or in the plural as "wait situations 204"), according to some embodiments of the present disclosure. In the example of Fig. 2, the first waiting situation 204(1) may be associated with an intersection that includes multiple traffic lights 206(1)-(2) (also referred to in the singular as "traffic light 206" or in the plural as "traffic lights 206"), with only two labeled for clarity, and multiple traffic lines 208(1)-(2) (also referred to in the singular as "traffic line 208" or in the plural as "traffic lines 208"), with only two labeled for clarity. Furthermore, the map 202 may associate the traffic lights 206 and / or the traffic lines 208 with the lanes 210(1)-(4) (also referred to in the singular as "lane 210" or in the plural as "lanes 210").For example, the map 202 may indicate that at least the first traffic light 206(1) is associated with the first lane 210(1), the second traffic light 206(2) is associated with the second lane 210(2), the third traffic light is associated with the third lane 210(3), and the fourth traffic light is associated with the fourth lane 210(4).

[0036] Furthermore, the second waiting situation 204(2) may be associated with an intersection that includes multiple stop signs 212(1)-(2) (also referred to in the singular as "stop sign 212" or in the plural as "stop signs 212") and multiple traffic lines 214(1)-(2) (also referred to in the singular as "traffic line 214" or in the plural as "traffic lines 214"), although only two are labeled for clarity. Furthermore, the map 202 may associate the stop signs 212 and / or the traffic lines 214 with the travel lanes 210. For example, the map 202 may indicate that at least the first stop sign 212(1) is associated with the third travel lane 210(3) and the second stop sign 212(2) is associated with the fourth travel lane 210(4).

[0037] Referring again to the example of Fig. 1, to determine wait situation information associated with a traffic feature, the wait situation component 102 may use a trip component 112 to retrieve one or more map streams 106 associated with the traffic feature. As described herein, a map stream 106 may be associated with the traffic feature based at least on a machine that generated the map stream 106 navigating along at least a portion of a road and / or a lane associated with the traffic feature. For a first example, if the traffic feature includes a stop sign, then the trip component 112 may determine that a map stream 106 in which a machine navigated along a road associated with the stop sign, such that the machine was deemed to stop at the stop sign, is associated with the stop sign.For a second example, if the traffic feature includes a traffic light associated with a lane of a road, then the trip component 112 may determine that a map stream 106 in which a machine navigated along the lane such that the machine was expected to stop at the traffic light when it was red is associated with the traffic light.

[0038] In some examples, the trip component 112 may perform one or more operations associated with the one or more identified map streams 106, such as cropping the one or more map streams 106. For example, and for a map stream 106, the trip component 112 may crop the map stream 106 to remove portions of the map stream 106 that are not associated with the traffic feature. As described herein, a portion of the map stream 106 may not be associated with the traffic feature based at least on the portion of the map stream 106 being outside a threshold distance from the traffic feature, such as 5 meters, 10 meters, 15 meters, and / or any other distance. In other words, after cropping, the map stream 106 may represent a trip (e.g., a path) that is within the threshold distance from the traffic feature.Thus, the trip component 112 generates and / or outputs trip data 114 representing trips (also referred to as “trip fragments” in some examples) that are more closely associated with the traffic feature.

[0039] For example, Fig. 3 illustrates an example of associating trips with traffic features, according to some embodiments of the present disclosure. As shown, the trip component 112 may associate a first trip 302(1) associated with a first machine 304(1) with the second traffic light 206(2), where the first trip 302(1) occurred at a first time, and a second trip 302(2) associated with a second machine 304(2) with the second traffic light 206(2), where the second trip 302(2) occurred at a second, different time. As shown, the first trip 302(1) may include at least one first stop location 306(1), and the second trip 302(2) may include at least one second stop location 306(2). In some examples, machines 304(1)-(2) may have stopped during trips 302(1)-(2) based on at least one state of second traffic light 206(2), where second traffic light 206(2) was red.

[0040] The trip component 112 may also associate a third trip 302(3), associated with the second machine 304(2), with the first stop sign 212(1). As shown, the third trip 302(3) may include at least one third stop location 306(3). In the example of Fig. 3, the trip component 112 may have tailored a mapstream associated with the second machine 304(2) to generate the second trip 302(2) starting at a location of the second machine 304(2) and ending at a location 308, and the third trip 302(3) starting at location 308 and ending at the end of the arrow.

[0041] Referring again to the example of Fig. 1, the process 100 may include the wait situation component 102 using a scoring component 116 to score the one or more trips represented by the trip data 114, where the one or more numerical values ​​are represented by scoring data 118. As described herein, in some examples, the scoring component 116 may score the one or more trips using one or more rules associated with the environment and / or traffic feature with which the one or more trips are associated, where the one or more rules may be represented by rule data 120 (which, in some examples, may also include and / or be associated with a portion of the map data 104).For a first example, if the traffic feature includes a stop sign, the one or more rules may include at least stopping at a location that is in front of the stop sign and / or within a threshold distance of the stop sign. For a second example, if the traffic feature includes a traffic light, the one or more rules may include proceeding if the traffic light is in a green state, yielding the right-of-way if the traffic light is in a yellow state, stopping if the traffic light is in a red state, following a direction associated with an arrow of the traffic light, and / or any other rule associated with traffic lights.

[0042] In some examples, the evaluation component 116 may provide a higher numerical value for trips that follow the one or more rules compared to trips that do not follow the one or more rules. For a first example, if the traffic feature includes a stop sign, the evaluation component 116 may provide a first numerical value for a first trip that includes a first machine decelerating as it approaches the stop sign, stopping before the stop sign, and then accelerating after the stop sign. Furthermore, the evaluation component 116 may provide a second numerical value for a second trip that includes a second machine decelerating as it approaches the stop sign but not coming to a complete stop before accelerating after the stop sign.In this example, the first numerical value may be greater than the second numerical value based at least on the first machine being more compliant with the one or more rules associated with the stop sign compared to the second machine.

[0043] For a second example, if the traffic feature includes a traffic light, the evaluation component 116 may provide a first numerical value for a first trip that includes a first machine decelerating when approaching the traffic light in a red state, stopping before the traffic light, and then accelerating past the traffic light when in a green state, in a direction associated with an arrow of the traffic light. Furthermore, the evaluation component 116 may provide a second numerical value for a second trip that includes a second machine decelerating when approaching the traffic light in a red state, stopping before the traffic light, and then accelerating past the traffic light when in a green state, but in a direction different from an arrow of the traffic light.In this example, the first numerical value may be greater than the second numerical value based at least on the first machine better following the one or more rules associated with the traffic light compared to the second machine because the first machine followed the direction of the traffic light arrow.

[0044] Furthermore, for a third example, and referring again to the example of Fig. 3, the evaluation component 116 may determine at least a first numerical value associated with the first trip 302(1) of the first machine 304(1), a second numerical value associated with the second trip 302(2) of the second machine 304(2), and a third numerical value associated with the third trip 302(3) of the second machine 304(2). Furthermore, the evaluation component 116 may determine the numerical values ​​for trips 302(1)-(3) (also referred to in the singular as "trip 302" or in the plural as "trips 302") as high because the machines 304(1)-(2) (also referred to in the singular as "machine 304" or in the plural as "machines 304") followed the rules associated with the environment and / or traffic characteristics.

[0045] Referring again to the example of Fig. 1 In some examples, the evaluation component 116 may preemptively discard one or more traffic features and / or one or more numerical values ​​based on at least one or more criteria. For example, the evaluation component 116 may discard traffic features and / or numerical values ​​based at least on a traffic feature not being associated with a map stream 106 (e.g., no machine has navigated within the portion of the environment associated with the traffic feature), the states associated with a traffic light being empty (e.g., the map streams 106 associated with the traffic light do not indicate the colors of the traffic light), the states associated with the traffic light meeting a threshold period (e.g.,equal to or greater than this), there is no overlap between the intervals of the traffic light states and the trips, the traffic light is not assigned to any states, and / or using other criteria. In other words, the evaluation component 116 may preemptively discard numerical values ​​for trips for which the evaluation component 116 may not be able to determine whether the machines have correctly followed the rules.

[0046] The process 100 may include the waiting situation component 102 using a grouping component 122 that assigns and / or groups the one or more trips associated with the traffic feature. For a first example, if the evaluation component 116 evaluates two trips for the same traffic feature, such as two trips navigating according to the rules associated with a stop sign, then the grouping component 122 may assign the two trips to each other and / or to the traffic feature.For a second example, if the evaluation component 116 evaluates the fourth trip for a road that includes two lanes associated with two different traffic lights, then the grouping component 122 may (1) associate two of the trips further associated with a first lane with each other and / or with a first of the traffic lights, and (2) associate two of the paths further associated with a second lane with each other and / or with a second of the traffic lights. In other words, the grouping component 122 may associate trips that at least partially overlap with respect to a lane within the environment and / or were influenced by the same traffic feature (e.g., the machines associated with the trips navigated according to the rules of the traffic feature).Thus, and as described in more detail herein, the wait situation component 102 is operable to use the one or more grouped trips to determine wait situation information associated with the traffic feature.

[0047] In some examples, the grouping component 122 may perform one or more processes to remove one or more of the one or more trips from the group. For example, the grouping component 122 may remove one or more trips associated with one or more numerical values ​​that do not meet (e.g., are less than) a threshold numerical value. For example, if the numerical values ​​are within a range, such as between 0 and 1, then the grouping component 122 may remove the one or more trips associated with one or more numerical values ​​that are less than 0.9 (e.g., the threshold numerical value).In some examples, the grouping component 122 may perform such processes because the one or more remote trips do not accurately represent how machines should navigate within the environment, i.e., by following the one or more rules associated with the traffic feature.

[0048] In some examples, the grouping component 122 may further perform one or more additional processes. For a first example, the grouping component 122 may merge the one or more trips to generate a final path associated with the traffic feature. In some examples, merging the one or more trips may include averaging the one or more trips, such as the poses, velocities, accelerations, stop locations, and / or the like. For a second example, the grouping component 122 may merge the one or more numerical values ​​to determine a final numerical value associated with the one or more trips and / or the final path.In some examples, merging the one or more numerical values ​​may include calculating the average of the one or more numerical values, the median of the one or more numerical values, the mode of the one or more numerical values, and / or performing another type of mathematical formula associated with the one or more numerical values. In any of the examples included herein, the grouping component 122 may then generate and / or output grouping data 124 representing the one or more trips, the final path, the one or more numerical values, and / or the final numerical value.

[0049] For example, Fig. 4 illustrates an example of grouping trips associated with a traffic feature located within an environment, according to some embodiments of the present disclosure. In the example of Fig. 4, the grouping component 122 may group the first trip 302(1) associated with the first machine 304(1) with the second trip 302(2) associated with the second machine 304(2) based at least on trips 302(1)-(2) being associated with the same lane 210(1) and / or trips 302(1)-(2) being associated with the same traffic signal 206(2). Therefore, in some examples, the grouping component 122 may combine the numerical values ​​associated with trips 302(1)-(2) to determine a final numerical value associated with the grouping. Furthermore, in some examples, the grouping component 122 may combine the trips 302(1)-(2) to determine a final path 402 associated with the grouping. As shown, the final path 402 may indicate at least one stop location 404, such as an average stop location, associated with the one or more trips 302(1)-(2).

[0050] Referring again to the example of Fig. 1, the process 100 may include the wait situation component 102 using a line component 126 to determine information for a wait situation associated with the traffic feature. As described herein, in some examples, the wait situation information may include a wait line representing a location within the environment for which machines should yield and / or stop when approaching the traffic feature and based at least on the rules associated with the traffic feature. For a first example, if the traffic feature includes a stop sign, then the wait line may indicate a location within the environment at which machines should stop when approaching the stop sign.For a second example, if the traffic feature includes a traffic light, then the wait line may specify a location within the environment at which machines should stop when approaching the traffic light and when the traffic light is in a specific state, such as a red light. For a third example, and again if the traffic feature includes a traffic light, then the wait line may specify a location within the environment at which machines should yield when approaching the traffic light and when the traffic light is in a specific state, such as a yellow light. Furthermore, for a fourth example, if the traffic feature includes a crosswalk light, then the wait line may specify a location within the environment at which machines should yield when approaching the crosswalk light and when the crosswalk light is in a specific state, such as flashing (e.g.B. indicates that people are crossing the zebra crossing).

[0051] As described herein, the line component 126 may use one or more techniques to determine the location of the wait line. For example, in some examples, the wait situation component 102 may use a candidate component 128 configured to determine one or more candidate lines that may be used for the wait line. In some examples, the candidate component 128 may determine that the candidate lines include one or more traffic lines that are within a threshold distance from the traffic feature, where the threshold distance is represented by threshold data 130. As described herein, a traffic line may include, without limitation, a stop line, a crosswalk line, a yield line, an intersection entry line, an intersection exit line, a turn line, a railroad crossing line, and / or any other line that may be located within the environment.Furthermore, a threshold distance may include, without limitation, 5 meters, 10 meters, 15 meters, and / or any other distance. In some examples, candidate component 128 may use the same threshold distance for all traffic features, while in other examples, candidate component 128 may use different threshold distances for different types of traffic features (e.g., a first threshold distance for traffic lights, a second threshold distance for stop signs, etc.).

[0052] For example, Fig. 5 illustrates an example of determining candidate lines associated with traffic features located within an environment, according to some embodiments of the present disclosure. As shown, the candidate component 128 may use various techniques to identify the candidate lines. For a first example, and with respect to the second traffic light 206(2), the candidate component 128 may use a threshold distance 502 from the second traffic light 206(2) to identify the candidate lines, which are the traffic lines 208 in the example of Fig. 5. For a second example, and with respect to the first stop sign 212(1), the candidate component 128 may use a threshold distance 504 in a first direction from the first stop sign 212(1) to identify candidate lines that include the traffic line 214(1). Furthermore, the candidate component 128 may use the threshold distance 504 in a second direction from the first stop sign 212(1) to identify candidate lines that include the traffic line 214(2) and another traffic line 506. In some examples, the candidate component 128 may use two threshold distances for the first stop sign 212(1) because stop lines may be located before or immediately after the stop signs based at least on where stop signs are placed.

[0053] Referring again to the example of Fig. 1, the line component 126 may receive candidate data 132 representing one or more candidate lines determined by the candidate component 128. The line component 126 may then use the grouping data 124 and / or the candidate data 132 to select a candidate line from the one or more candidate lines to use as the waiting line. For example, in some examples, such as when the grouping data 124 represents the one or more trips associated with the traffic feature, the line component 126 may select the candidate line at which a majority of the one or more trips within the vicinity stop.For example, if the grouping data 124 represents five trips and four of the trips stop at a first candidate line while one of the trips stops at a second candidate line, then the line component 126 may select the first candidate line as the waiting line for the traffic feature. In some examples, such as when the grouping data 124 again represents the one or more trips associated with the traffic feature, the line component 126 may select the candidate line at which a threshold number (e.g., one, five, ten, fifty, one hundred, etc.) of the one or more trips stop within the vicinity.For example, if the grouping data 124 represents five trips, four of the trips stop at a first candidate line, one of the trips stops at a second candidate line, and the threshold number is three trips, then the line component 126 may again select the first candidate line as the waiting line for the traffic feature.

[0054] In some examples, such as when the one or more trips are merged to generate a final path associated with the traffic feature, the line component 126 may select the candidate line at which the final path stops within the environment. For example, if the candidate data 132 represents two candidate lines, then the line component 126 may select the line at which the final path stops (and / or is closer to it) as the hold line for the traffic feature. While these are just some example techniques for how the line component 126 may select a candidate line from among one or more candidate lines to include as a hold line for a traffic feature, in other examples, the line component 126 may use additional and / or alternative techniques.

[0055] For example, and referring again to the example of Fig. 4, in some examples, the line component 126 may select the second traffic line 208(2) to be included as the waiting line for the second traffic signal 206(2) based at least on a majority of the trips 302 (and / or a threshold number of trips 302) stopping at the second traffic line 208(2). Additionally or alternatively, in some examples, the line component 126 may again select the second traffic line 208(2) to be included as the waiting line for the second traffic signal 206(2) based at least on the end path 402 stopping at the second traffic line 208(2). Furthermore, in some examples, the line component 126 may select the first traffic line 214(1) to be included as the waiting line for the first stop sign 212(1) based at least on a majority of the trips that only include the third trip 302(3) in the example of Fig. 4, stops at the first traffic line 214(1).

[0056] Referring again to the example of Fig. 1, in some examples, the line component 126 may use additional and / or alternative techniques to determine a location of the waiting line within the environment. For example, if the candidate component 128 is unable to determine candidate lines associated with the traffic feature (e.g., there are no traffic lines within the environment that are within a threshold distance of the traffic feature), then the line component 126 may project a waiting line into the environment. For example, in some examples, such as when the grouping data 124 represents the one or more trips associated with the traffic feature, the line component 126 may project a waiting line at a location within the environment where a majority of the one or more trips within the environment stop.For example, if the grouping data 124 represents five trips and four of the trips stop at a first location while one of the trips stops at a second location, then the line component 126 may project the wait line for the traffic feature at the first location.

[0057] In some examples, such as when the grouping data 124 again represents the one or more trips associated with the traffic feature, the line component 126 may project a wait line at a location where a threshold number (e.g., one, five, ten, fifty, one hundred, etc.) of the one or more trips stop within the vicinity. For example, if the grouping data 124 represents five trips, four of the trips stop at a first location, one of the trips stops at a second location, and the threshold number is three trips, then the line component 126 may again project the wait line for the traffic feature at the first location. In some examples, such asWhen the one or more trips are merged to generate a final path associated with the traffic feature, the line component 126 may project a wait line at a location where the final path stops within the environment. While these are only some example techniques for how the line component 126 may determine a location of a projected wait line for a traffic feature, the line component 126 may use additional and / or alternative techniques in other examples.

[0058] For example, Fig. 6 illustrates an example of using trips to determine a location for projecting a wait line within an environment, according to some embodiments of the present disclosure. As shown, a map 602 may be similar to map 202, however, the second travel lane 210(2) may no longer include the traffic lines 208(1)-(2) associated with the second traffic signal 206(2). Therefore, the line component 126 may use trips 604(1)-(2) associated with machines 606(1)-(2) that occurred at different times to determine locations 608(1)-(2) associated with the second traffic signal 206(2). The line component 126 may then project a wait line 610 within the environment, with the wait line 610 located at the stop locations 608(1)-(2).

[0059] Referring again to the example of Fig. 1, the process 100 may include the line component 126 generating and / or outputting updated wait situation data 132 representing the wait situation information, such as the determined wait line. The wait situation component 102 may then update the map data 104 to include the updated wait situation data 132. Furthermore, in some examples, the process 100 may further repeat to determine wait situation information for any number of traffic features and / or wait situations within the environment.For example, the wait situation component 102 may perform these processes to determine wait situation information associated with the first traffic light 206(1) associated with the first lane 210(1), the second traffic light 206(2) associated with the second lane 210(2), the third traffic light associated with the third lane 210(3), the fourth traffic light associated with the fourth lane 210(4), the first stop sign 212(1) associated with the third lane 210(3), and the second stop sign 212(2) associated with the fourth lane 210(4).

[0060] In some examples, the wait situation component 102 may perform additional processes when determining the wait situation information. For example, the wait situation component 102 may group traffic features together, such as traffic features associated with the same wait situation. For example, the wait situation component 102 may group traffic features associated with the same intersection and then determine the wait situation information associated with the group of traffic features using one or more of the processes described herein.

[0061] Therefore, the wait situation component 102 is in performing the process 100 of Fig. 1 is capable of updating the map data 104 to indicate additional waiting situation information, such as the waiting lines for yielding the right of way and / or stopping at various traffic features. For example, Fig. 7 illustrates an example of map 202 updated to indicate wait lines associated with traffic features, in accordance with some embodiments of the present disclosure.

[0062] As shown, the map 202 may now indicate a first wait line 702(1) associated with the first traffic light 206(1) and / or the first lane 210(1), a second wait line 702(2) associated with the second traffic light 206(2) and / or the second lane 210(2), a third wait line 702(3) associated with the third traffic light and / or the third lane 210(3), a fourth wait line 702(4) associated with the fourth traffic light and / or the fourth lane 210(4), a fifth wait line 702(5) associated with the first stop sign 212(1) and / or the third lane 210(3), and a sixth wait line 702(6) associated with the second stop sign 212(2) and / or the fourth lane 210(4).In other words, by performing process 100, the wait situation component 102 is able to determine the locations of the wait lines 702(1)-(6) using the traffic lines and then update the map 202 to indicate the locations.

[0063] Referring again to the example in Fig. 1, the process 100 may include sending the map data 104, after updating, to one or more machines 136 located within the environment. Thus, the one or more machines 136 may use the map data 104 when navigating within the environment, such as to determine where to yield and / or stop in relation to traffic features. For a first example, if a machine 136 is approaching a stop sign within the environment, the machine 136 may use the updated map to determine to stop at the hold line associated with the stop sign. For a second example, if a machine 136 is approaching a traffic light that is in a red state, the machine 136 may use the updated map to determine to stop at the hold line associated with the traffic light.Furthermore, for a third example, if a machine 136 approaches a crosswalk that includes a light signal, and the light signal is in a flashing state indicating that people are crossing the crosswalk, then the machine 136 may use the updated map to determine to stop at the hold line associated with the crosswalk.

[0064] With reference to Fig. 8 and Fig. 9, each block of the methods 800 and 900 described herein comprises a computational process that may be performed using any combination of hardware, firmware, and / or software. Various functions may be performed, for example, by a processor executing instructions stored in memory. The methods 800 and 900 may also be embodied as computer-usable instructions stored on computer storage media. The methods 800 and 900 may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. Additionally, the methods 800 and 900 are described by way of example with respect to Fig. 1. However, these methods 800 and 900 may additionally or alternatively be performed by any system or combination of systems, including, but not limited to, the systems described herein.

[0065] Fig. 8 illustrates a flowchart showing a method 800 for determining wait situation information associated with a traffic feature, according to some embodiments of the present disclosure. The method 800 may include, at block B802, obtaining data representing one or more trips associated with a traffic feature located within an environment. For example, the wait situation component 102 may receive the map streams 106 associated with the environment, where the map streams 106 include the data representing the one or more trips. In some examples, the wait situation component 102 (e.g., the trip component 112) may trim the one or more trips to include one or more fragments associated with the traffic feature, where the one or more fragments may also be referred to as one or more "paths."In some examples, the waiting situation component 102 (e.g., the evaluation component 116) may determine one or more numerical values ​​associated with the one or more trips, e.g., based on how well the one or more trips followed one or more rules associated with the environment and / or the traffic feature.

[0066] The process 800 may include, at block B804, determining, based at least on map data, one or more candidate lines associated with the traffic feature. For example, the wait situation component 102 (e.g., the candidate component 128) may use the map data 104 to determine the one or more candidate lines. As described herein, in some examples, the wait situation component 102 may identify the one or more candidate lines based at least on the one or more candidate lines being within a threshold distance from the traffic feature, being associated with a same lane as the traffic feature, being associated with the traffic feature, and / or using another technique.

[0067] The process 800 may include, in block B806, determining, based on at least the one or more trips, that a candidate line of the one or more candidate lines includes a wait line for the traffic feature. For example, the wait situation component 102 (e.g., the line component 126) may select a candidate line from the one or more candidate lines as the wait line. As described herein, in some examples, the wait situation component 102 may use one or more techniques to select the candidate line. For a first example, the wait situation component 102 may select the candidate line for which a majority of the one or more trips stop when approaching the traffic feature. For a second example, such as when the wait situation component 102 (e.g.,the grouping component 122) which combines one or more trips to generate a final path, the waiting situation component 102 selects the candidate line for which the final path stops when approaching the traffic feature.

[0068] The method 800 may include, at block B808, updating the map data to indicate that the candidate line includes the wait line for the traffic feature. For example, the wait situation component 102 may use updated wait situation data 134 to update the map data 104 to indicate the new wait situation information associated with the traffic feature. As described herein in some examples, the map data 104 may be updated to indicate that the traffic line containing the selected candidate line is also associated with the wait line for the traffic feature.

[0069] Fig. 9 illustrates a flowchart showing a method 900 for determining a wait line for a traffic feature associated with a wait situation, according to some embodiments of the present disclosure. The method 900 may include, at block B902, obtaining data representing one or more trips associated with a traffic feature located within an environment. For example, the wait situation component 102 may receive the map streams 106 associated with the environment, where the map streams 106 include the data representing the one or more trips. In some examples, the wait situation component 102 (e.g., the trip component 112) may trim the one or more trips to include one or more fragments associated with the traffic feature, where the one or more fragments may also be referred to as one or more "paths."

[0070] The method 900 may include, at block B904, determining, based on at least one or more rules associated with the traffic feature, one or more numerical values ​​associated with the one or more trips. For example, the wait situation component 102 (e.g., the evaluation component 116) may determine the one or more numerical values ​​associated with the one or more trips. As described herein, the wait situation component 102 may determine the one or more numerical values ​​based at least on how well the one or more trips followed the one or more rules associated with the traffic feature. In some examples, the one or more numerical values ​​may be associated with a range, such as between 0 and 1 (and / or another range).

[0071] The method 900 may include, at block B906, determining, based on at least the one or more numerical values, a group of trips that includes at least a portion of the one or more trips. For example, the wait situation component 102 (e.g., the grouping component 122) may group the one or more trips based on at least the one or more numerical values. As described herein, in some examples, the wait situation component 102 may discard one or more of the one or more trips, such as one or more of the one or more trips associated with one or more numerical values ​​that do not meet (e.g., are less than) a threshold numerical value. Furthermore, in some examples, the wait situation component 102 may merge the one or more trips to generate a final path associated with the traffic feature.

[0072] The method 900 may include, at block B908, determining, based at least on the group of trips, a stop location associated with the traffic feature. For example, the wait situation component 102 (e.g., the line component 126) may determine the stop location based at least on the group of trips. As described herein, in some examples, the wait situation component 102 may determine the stop location as associated with a candidate line from the one or more candidate lines associated with the traffic feature. In some examples, the wait situation component 102 may determine the stop location based at least on a majority of the one or more trips and / or a merged final path stopping at the stop location.

[0073] The method 900 may include, at block B910, determining, based at least on the stop location, a wait line associated with the traffic feature. For example, the wait situation component 102 (e.g., the line component 126) may determine the wait line based at least on the stop location. For example, in some examples, if the stop location is associated with a candidate line, the wait situation component 102 may then determine that the candidate line includes the wait line. In some examples, if the stop location is not associated with a candidate line, the wait situation component 102 may then project the wait line at the stop location. In each example, the wait situation component 102 may then update the map data 104 to indicate the wait line. EXEMPLARY AUTONOMOUS VEHICLE

[0074] Fig. 10A is an illustration of an exemplary autonomous vehicle 1000, according to some embodiments of the present disclosure. The autonomous vehicle 1000 (alternatively referred to herein as "vehicle 1000") may include, without limitation, a passenger vehicle, such as a car, a truck, a bus, an emergency service vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire engine, a police vehicle, an ambulance, a boat, a construction vehicle, an underwater vehicle, a robotic vehicle, a drone, an aircraft, a vehicle coupled to a trailer (e.g., a semi-trailer used to transport cargo), and / or another type of vehicle (e.g., that is unmanned and / or accommodates one or more passengers).Autonomous vehicles are generally described in terms of levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) standard "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and prior and future versions of this standard). Vehicle 1000 may exhibit functionality consistent with one or more of the Level 3 through Level 5 autonomous driving levels.The vehicle 1000 may have functionality according to one or more of Level 1 through Level 5 of autonomous driving levels. For example, depending on the embodiment, the vehicle 1000 may be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term "autonomous" as used herein may include any and / or all types of autonomy for the vehicle 1000 or other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, assisted autonomy, semi-autonomous, primarily autonomous, or another designation.

[0075] The vehicle 1000 may include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle 1000 may include a propulsion system 1050, such as an internal combustion engine, a hybrid electric power plant, a pure electric motor, and / or another type of propulsion system. The propulsion system 1050 may be connected to a drivetrain of the vehicle 1000, which may include a transmission to enable propulsion of the vehicle 1000. The propulsion system 1050 may be controlled in response to receiving signals from the throttle or accelerator 1052.

[0076] A steering system 1054, which may include a steering wheel, may be used to steer the vehicle 1000 (e.g., along a desired path or route) when the propulsion system 1050 is operating (e.g., when the vehicle is moving). The steering system 1054 may receive signals from a steering actuator 1056. The steering wheel may be optional for full automation (Level 5).

[0077] The brake sensor system 1046 may be used to apply the vehicle brakes in response to receiving signals from the brake actuators 1048 and / or the brake sensors.

[0078] The one or more controllers 1036 that control one or more systems on chips (SoCs) 1004 ( Fig. 10C) and / or GPUs may provide signals (e.g., representative of instructions) to one or more components and / or systems of the vehicle 1000. For example, the one or more controllers may send signals to apply the vehicle brakes via one or more brake actuators 1048, to apply the steering system 1054 via one or more steering actuators 1056, to apply the propulsion system 1050 via one or more throttle / accelerator devices 1052. The one or more controllers 1036 may include one or more built-in (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 1000.The one or more controllers 1036 may include a first controller 1036 for autonomous driving functions, a second controller 1036 for functional safety functions, a third controller 1036 for artificial intelligence functions (e.g., computer vision), a fourth controller 1036 for infotainment functions, a fifth controller 1036 for emergency redundancy, and / or other controllers. In some examples, a single controller 1036 may perform two or more of the above functionalities, two or more controllers 1036 may perform a single functionality, and / or any combination thereof.

[0079] The one or more controllers 1036 may provide the signals to control one or more components and / or systems of the vehicle 1000 in response to sensor data received from one or more sensors (e.g., sensor inputs). The sensor data may be received, for example and without limitation, from one or more of the following: global navigation satellite systems (“GNSS”) sensor(s) 1058 (e.g., global positioning system sensor(s)), RADAR sensor(s) 1060, ultrasonic sensor(s) 1062, LIDAR sensor(s) 1064, inertial measurement unit (IMU) sensor(s) 1066 (e.g., accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s), etc.), microphone(s) 1096, stereo camera(s) 1068, wide-angle camera(s) 1070 (e.g., fisheye cameras), infrared camera(s) 1072, ambient camera(s) 1074 (e.g.,360-degree cameras), long-range and / or medium-range camera(s) 1098, speed sensor(s) 1044 (e.g., for measuring the speed of the vehicle 1000), vibration sensor(s) 1042, steering sensor(s) 1040, brake sensor(s) (e.g., as part of the brake sensor system 1046), and / or other sensor types.

[0080] One or more of the controllers 1036 may receive inputs (e.g., in the form of input data) from an instrument cluster 1032 of the vehicle 1000 and provide outputs (e.g., in the form of output data, display data, etc.) via a human-machine interface (HMI) display 1034, an audible annunciator, a speaker, and / or via other components of the vehicle 1000. The outputs may include information such as vehicle speed, RPM, time, map data (e.g., the high-definition (“HD”) map 1022 of Fig. 10C), location data (e.g., the location of the vehicle 1000, e.g., on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the one or more controllers 1036, etc. For example, the HMI display 1034 may display information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing light signal, etc.) and / or information about maneuvers that the vehicle has performed, is currently performing, or will perform (e.g., change lanes now, take exit 34B in two miles, etc.).

[0081] The vehicle 1000 further includes a network interface 1024 that may utilize one or more wireless antennas 1026 and / or modems for communication over one or more networks. The network interface 1024 may, for example, be capable of communication via Long-Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communication (GSM), IMT-CDMA Multi-Carrier (CDMA2000), etc. The one or more wireless antennas 1026 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc.and / or Low Power Wide Area Networks (LPWANs), such as LoRaWAN, SigFox, etc.

[0082] Fig. 10B is an example of camera locations and fields of view for the example autonomous vehicle 1000 of Fig. 10A, according to some embodiments of the present disclosure. The cameras and respective fields of view are an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at various locations on the vehicle 1000.

[0083] The camera types for the cameras may include, but are not limited to, digital cameras that may be configured for use with the components and / or systems of the vehicle 1000. The one or more cameras may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera types may be capable of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The cameras may use rolling shutters, global shutters, another type of shutter, or a combination thereof.In some examples, the color filter array may include a red clear clear clear (RCCC) color filter array, a red clear clear blue (RCCB) color filter array, a red blue green clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor color filter array (RGGB), a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear-pixel cameras, such as cameras with an RCCC, an RCCB, and / or an RBGC color filter array, may be used to increase light sensitivity.

[0084] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multifunction mono camera can be installed to provide features including lane departure warning, traffic sign assist, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0085] One or more of the cameras may be mounted in a bracket, such as a specially designed (three-dimensional ("3D") printed) bracket, to eliminate stray light and reflections from the vehicle interior (e.g., dashboard reflections reflected in the windshield) that could interfere with the camera's image data acquisition. With regard to the bracket for exterior mirrors, the exterior mirrors may be custom 3D printed so that the camera mounting plate is adapted to the shape of the exterior mirror. In some examples, the one or more cameras may be integrated into the exterior mirror. For side-mounted cameras, the one or more cameras may also be integrated into the four pillars at each corner of the cabin.

[0086] Cameras with a field of view that includes portions of the environment in front of the vehicle 1000 (e.g., forward-facing cameras) can be used for the surrounding view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 1036 and / or control SoCs, to provide information critical for establishing an occupancy grid and / or determining preferred vehicle paths. Forward-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS features and systems that include lane departure warnings (LDW), autonomous cruise control (ACC), and / or other features such as traffic sign detection.

[0087] A variety of cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform that includes a complementary metal oxide semiconductor (CMOS) color imager. Another example is the wide-angle cameras 1070, which can be used to capture objects that enter the field of view from the periphery (e.g., pedestrians, crossing vehicles, or bicycles). Although Fig. 10B illustrates only one wide-angle camera, any number (including zero) of wide-angle cameras 1070 may be present on the vehicle 1000. Furthermore, any number of long-range cameras 1098 (e.g., a long-range stereo camera pair) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The one or more long-range cameras 1098 may also be used for object detection and classification, as well as basic object tracking.

[0088] Any number of stereo cameras 1068 may also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo cameras 1068 may include an integrated control unit comprising a scalable processing unit that can provide a programmable logic ("FPGA") and a multi-core microprocessor with an integrated controller area network ("CAN") or Ethernet interface on a single chip. Such a unit can be used to create a 3D map of the vehicle's surroundings that includes a distance estimate for all points in the image. One or more alternative stereo cameras 1068 may include a compact stereo vision sensor that can include two camera lenses (one each on the left and right) and an image processing chip that can measure the distance between the vehicle and the target object and process the generated information (e.g.,metadata) to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1068 may be used in addition to or alternatively to those described here.

[0089] Cameras with a field of view that includes portions of the environment to the side of the vehicle 1000 (e.g., side cameras) may be used for the environment view and provide information used to create and update the occupancy grid and to generate collision warnings in the event of a side impact. For example, the one or more environment cameras 1074 (e.g., four environment cameras 1074, as in Fig. 10B) may be positioned on the vehicle 1000. The one or more surround cameras 1074 may include one or more wide-angle cameras 1070, one or more fisheye cameras, one or more 360-degree cameras, and / or the like. For example, four fisheye cameras may be mounted on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may utilize three surround cameras 1074 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround camera.

[0090] Cameras with a field of view that includes portions of the environment behind the vehicle 1000 (e.g., rearview cameras) may be used for parking assistance, surround view, rear impact warnings, and occupancy grid creation and updating. A variety of cameras may be used, including, but not limited to, cameras that are also suitable as one or more forward-facing cameras (e.g., one or more long-range and / or medium-range cameras 1098, one or more stereo cameras 1068, one or more infrared cameras 1072, etc.), as described herein.

[0091] Fig. 10C is a block diagram of an example system architecture for the example autonomous vehicle 1000 of Fig. 10A, according to some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., engines, interfaces, functions, arrangements, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions performed by entities described herein may be performed by hardware, firmware, and / or software. Various functions may be performed, for example, by a processor executing instructions stored in memory.

[0092] Each of the vehicle’s components, features and systems 1000 in Fig. 10C is illustrated as being connected via bus 1002. Bus 1002 may include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). A CAN may be a network within vehicle 1000 that serves to support the control of various features and functions of vehicle 1000, such as the application of brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus may be read to determine steering wheel angle, vehicle speed, engine speed (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0093] Although bus 1002 is described herein as a CAN bus, this is not intended to be a limitation. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to the CAN bus. Furthermore, while a single wire is used to represent bus 1002, this is not intended to be a limitation. For example, there may be any number of buses 1002, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using a different protocol. In some examples, two or more buses 1002 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1002 may be used for collision avoidance functionality and a second bus 1002 may be used for actuation control.In each example, each bus 1002 may communicate with one of the components of the vehicle 1000, and two or more buses 1002 may communicate with the same components. In some examples, each SoC 1004, each controller 1036, and / or each computer within the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle 1000) and may be connected to a common bus, such as the CAN bus.

[0094] The vehicle 1000 may include one or more controllers 1036 as described herein with reference to Fig. 10A. The one or more controllers 1036 may be used for a variety of functions. The one or more controllers 1036 may be coupled to any of the various other components and systems of the vehicle 1000 and may be used for control of the vehicle 1000, artificial intelligence of the vehicle 1000, infotainment for the vehicle 1000, and / or the like.

[0095] The vehicle 1000 may include one or more systems on a chip (SoC) 1004. The SoC 1004 may include one or more CPUs 1006, one or more GPUs 1008, one or more processors 1010, one or more caches 1012, one or more accelerators 1014, one or more data memories 1016, and / or other components and features not illustrated. The one or more SoCs 1004 may be used to control the vehicle 1000 in a variety of platforms and systems. For example, the one or more SoCs 1004 may be combined in a system (e.g., the system of the vehicle 1000) with an HD card 1022 that may be accessed via a network interface 1024 from one or more servers (e.g., the one or more servers 1078 of Fig. 10D) Receive map refreshes and / or updates.

[0096] The one or more CPUs 1006 may include a CPU cluster or CPU complex (alternatively referred to herein as a "CCPLEX"). The one or more CPUs 1006 may include multiple cores and / or L2 caches. For example, in some embodiments, the one or more CPUs 1006 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the one or more CPUs 1006 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The one or more CPUs 1006 (e.g., the CCPLEX) may be configured to support concurrent operation of clusters, such that any combination of the clusters of the one or more CPUs 1006 may be active at any given time.

[0097] The one or more CPUs 1006 may implement power management functions that include one or more of the following features: individual hardware blocks may be automatically clock controlled when idle to conserve dynamic power; each core clock may be controlled when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power controlled; each core cluster may be independently clock controlled if all cores are clock controlled or power controlled; and / or each core cluster may be independently power controlled if all cores are power controlled.The one or more CPUs 1006 may further implement an enhanced power state management algorithm, where permissible power states and expected wake-up times are determined, and the hardware / microcode determines the best power state to enter for the core, cluster, and CCPLEX. The processing cores may support simplified power state entry sequences in software, offloading the work to the microcode.

[0098] The one or more GPUs 1008 may include an integrated GPU (alternatively referred to herein as an "iGPU"). The one or more GPUs 1008 may be programmable and may be efficient for parallel workloads. The one or more GPUs 1008 may, in some examples, use an extended Tensor instruction set. The one or more GPUs 1008 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of memory capacity), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512 KB of memory capacity). In some embodiments, the one or more GPUs 1008 may include at least eight streaming microprocessors. The one or more GPUs 1008 may use one or more application programming interfaces (APIs) for computations.In addition, the one or more GPUs 1008 may use one or more parallel computing platforms and / or programming models (e.g., CUDA from NVIDIA).

[0099] The one or more GPUs 1008 may be power-optimized for best performance in automotive and embedded use cases. For example, the one or more GPUs 1008 may be fabricated on a fin field-effect transistor (FinFET). However, this is not a limitation, and the one or more GPUs 1008 may also be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor may include a number of mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks.In such an example, each processing block can be assigned 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64 KB register file. Furthermore, the streaming microprocessors can include independent parallel integer and floating-point datapaths to enable efficient execution of workloads with a mix of computations and addressing calculations. The streaming microprocessors can include independent thread scheduling to enable fine-grained synchronization and cooperation between parallel threads. The streaming microprocessors can include a combined L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0100] The one or more GPUs 1008 may include high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem to provide, in some examples, a peak memory bandwidth of approximately 900 GB / second. In some examples, synchronous graphics random access memory (SGRAM), such as double data rate type five (GDDR5), may be used in addition to or alternatively to the HBM memory.

[0101] The one or more GPUs 1008 may include unified memory technology that includes access counters to enable more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for processor-shared memory areas. In some examples, Address Translation Services (ATS) support may be used to allow the one or more GPUs 1008 to directly access the page tables of the one or more CPUs 1006. In such examples, if the Memory Management Unit (MMU) of the one or more GPUs 1008 fails, an address translation request may be sent to the one or more CPUs 1006.In response, the one or more CPUs 1006 may look up the virtual-to-physical mapping for the address in their page tables and send the translation back to the one or more GPUs 1008. Thus, the unified memory technology may enable a single unified virtual address space for the memory of both the one or more CPUs 1006 and the one or more GPUs 1008, thereby simplifying the programming of the one or more GPUs 1008 and the porting of applications to the one or more GPUs 1008.

[0102] Additionally, the one or more GPUs 1008 may include an access counter that may track the frequency of access by the one or more GPUs 1008 to memory of other processors. The access counter may help move memory pages to the physical memory of the processor that accesses the pages most frequently.

[0103] The one or more SoCs 1004 may include any number of caches 1012, including those described herein. For example, the one or more caches 1012 may include an L3 cache available to both the one or more CPUs 1006 and the one or more GPUs 1008 (e.g., connected to both the one or more CPUs 1006 and the one or more GPUs 1008). The one or more caches 1012 may include a write-back cache that may track line states, e.g., by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may also be used.

[0104] The one or more SoCs 1004 may include one or more arithmetic logic units (ALUs) that may be utilized in performing processing related to any of the various tasks or operations of the vehicle 1000, such as processing DNNs. Additionally, the one or more SoCs 1004 may include one or more floating-point units (FPUs)—or other mathematical co-processors or numerical co-processors—for performing mathematical operations within the system. For example, the one or more SoCs 1004 may include one or more FPUs integrated as execution units within one or more CPUs 1006 and / or one or more GPUs 1008.

[0105] The one or more SoCs 1004 may include one or more accelerators 1014 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the one or more SoCs 1004 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the one or more GPUs 1008 and offload some of the tasks of the one or more GPUs 1008 (e.g., to free up more cycles of the one or more GPUs 1008 to perform other tasks). For example, the one or more accelerators 1014 may be configured for targeted workloads (e.g.,Perception, convolutional neural networks (CNNs), etc.) that are robust enough to be suitable for acceleration can be used. The term "CNN" as used here can include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0106] The one or more accelerators 1014 (e.g., the hardware acceleration cluster) may include a deep learning accelerator (DLA). The one or more DLAs may include one or more tensor processing units (TPUs) configured to provide an additional tens of trillion operations per second for deep learning applications and inferencing. The TPUs may be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). The one or more DLAs may also be optimized for a specific set of neural network types and floating-point operations, as well as for inferencing. The design of the one or more DLAs may provide more performance per millimeter than a general-purpose GPU, far exceeding the performance of a CPU.The one or more TPUs can perform multiple functions, including a single-instance convolution function that supports, for example, INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0107] The one or more DLAs can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for a variety of functions, including, for example and without limitation: a CNN for object identification and detection using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety and / or security events.

[0108] The one or more DLAs can perform any function of the one or more GPUs 1008, and by using an inference accelerator, a developer can, for example, dedicate either the one or more DLAs or the one or more GPUs 1008 to each function. For example, the developer can concentrate the processing of CNNs and floating-point operations on the one or more DLAs and leave other functions to the one or more GPUs 1008 and / or other accelerators 1014.

[0109] The one or more accelerators 1014 (e.g., the hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may also be referred to herein as a computer vision accelerator. The one or more PVAs may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The one or more PVAs may provide a balance between performance and flexibility. For example, and without limitation, each PVA may include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA) cores, and / or any number of vector processors.

[0110] The RISC cores may interact with image sensors (e.g., the image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of the RISC cores may include any amount of memory. The RISC cores may use any number of protocols, depending on the embodiment. In some examples, the RISC cores may execute a real-time operating system (RTOS). The RISC cores may be implemented with one or more integrated circuits, application-specific integrated circuits (ASICs), and / or memory devices. The RISC cores may include, for example, an instruction cache and / or tightly coupled RAM.

[0111] The DMA may enable components of the PVA(s) to access the system's memory independently of the one or more CPUs 1006. The DMA may support any number of features designed to optimize the PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In some examples, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0112] The vector processors may be programmable processors that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing functions. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or memory (e.g., VMEM).A VPU core can contain a digital signal processor, such as a single instruction multiple data (SIMD) and very long instruction word (VLIW) digital signal processor. Combining SIMD and VLIW can increase throughput and speed.

[0113] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each of the vector processors may be configured to operate independently of the other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, the multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may concurrently execute different computer vision algorithms on the same image, or even execute different algorithms on consecutive images or portions of an image.Among other things, any number of PVAs can be included in the hardware acceleration cluster, and any number of vector processors can be included in each of the PVAs. Furthermore, the one or more PVAs can contain additional memory for error correcting code (ECC) to increase the overall security of the system.

[0114] The one or more accelerators 1014 (e.g., the hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM to the one or more accelerators 1014. In some examples, the on-chip memory may include at least 4 MB of SRAM, consisting of, for example, and without limitation, eight field-configurable memory blocks accessible by both the PVA and the DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and the DLA may access the memory through a backbone that provides high-speed access to the memory to the PVA and the DLA.The backbone may include an on-chip computer vision network connecting the PVA and DLA to the memory (e.g., using the APB).

[0115] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA are delivering ready and valid signals before transmitting control signals / addresses / data. Such an interface can provide separate phases and channels for transmitting control signals / addresses / data, as well as bursty communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508, although other standards and protocols can also be used.

[0116] In some examples, the one or more SoCs 1004 may include a real-time ray tracing hardware accelerator, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the positions and extents of objects (e.g., within a world model) to generate real-time visualization simulations, for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for simulation of SONAR systems, for general wave propagation simulation, for comparison with LIDAR data for localization, and / or for other functions, and / or for other purposes. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more operations related to ray tracing.

[0117] The one or more accelerators 1014 (e.g., the hardware accelerator cluster) have a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for critical processing steps in ADAS and autonomous vehicles. The capabilities of the PVA are well suited to algorithmic areas that require predictable processing with low power and low latency. In other words, the PVA is well suited for semi-dense or dense regular computations, even on small datasets, that require predictable runtimes with low latency and low power. In the context of autonomous vehicle platforms, the PVAs are therefore designed to execute classical computer vision algorithms because they are efficient at object detection and operate on integer mathematics.

[0118] According to one embodiment of the technology, the PVA is used, for example, to perform computer stereovision. In some examples, a semi-global matching-based algorithm may be used, although this is not intended as a limitation. Many applications for Level 3-5 autonomous driving require on-the-fly motion estimation or stereo matching (e.g., structure from motion, pedestrian detection, lane detection, etc.). The PVA can perform a computer stereovision function on inputs from two monocular cameras.

[0119] In some examples, the PVA can be used to perform dense optical flow, such as processing raw radar data (e.g., using a 4D Fast Fourier Transform) to provide processed radar. In other examples, the PVA is used for time-of-flight depth processing, such as processing raw time-of-flight data to provide processed time-of-flight data.

[0120] The DLA can be used to power any type of network to improve control and driving safety; for example, this includes a neural network that outputs a confidence measure for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative "weight" to each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only those detections that exceed the threshold as true positives.In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most certain detections should be considered as triggers for AEB. The DLA may employ a neural network to regress the confidence value. The neural network may take as input at least a subset of parameters, such as, but not limited to, the bounding box dimensions, the ground plane estimate obtained (e.g., from another subsystem), the output of the inertial measurement unit (IMU) sensor 1066 correlated with the orientation of the vehicle 1000, distance, 3D position estimates of the object obtained by the neural network and / or other sensors (e.g., one or more LIDAR sensors 1064 or one or more RADAR sensors 1060).

[0121] The one or more SoCs 1004 may include the one or more data stores 1016 (e.g., memory). The one or more data stores 1016 may be on-chip memory on the one or more SoCs 1004 in which neural networks to be executed on the GPU and / or the DLA may be stored. In some examples, the one or more data stores 1016 may be large enough to store multiple instances of neural networks for redundancy and security. The one or more data stores 1012 may include one or more L2 or L3 caches 1012. The reference to the one or more data stores 1016 may include a reference to the memory associated with the PVA, the DLA, and / or one or more other accelerators 1014, as described herein.

[0122] The one or more SoCs 1004 may include one or more processors 1010 (e.g., embedded processors). The one or more processors 1010 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle boot power and management functions and associated security enforcement. The boot and power management processor may be part of the boot sequence of the one or more SoCs 1004 and may provide runtime power management services. The boot and power management processor may provide clock and voltage programming, assisting with system transitions to a low-power state, managing the thermals and temperature sensors of the one or more SoCs 1004, and / or managing the one or more SoCs 1004 power states.Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the one or more SoCs 1004 may use the ring oscillators to sense the temperatures of the one or more CPUs 1006, the one or more GPUs 1008, and / or the one or more accelerators 1014. If it is determined that the temperatures exceed a threshold, the boot and power management processor may enter a temperature fault routine and place the one or more SoCs 1004 into a lower power state and / or place the vehicle 1000 into a chauffeur-to-safe-stop mode (e.g., bring the vehicle 1000 to a safe stop).

[0123] The one or more processors 1010 may also include a set of embedded processors that can serve as an audio processing engine. The audio processing engine may be an audio subsystem that enables full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0124] The one or more processors 1010 may also include an always-on processor engine that may provide the necessary hardware functions to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0125] The one or more processors 1010 may also include a safety cluster engine containing a dedicated processor subsystem for safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, an interrupt controller, etc.), and / or routing logic. In a safety mode, the two or more cores may operate in a lockstep mode, functioning as a single core with comparison logic that captures any differences between their operations.

[0126] The one or more processors 1010 may also include a real-time camera engine, which may include a dedicated processor subsystem for managing the real-time camera.

[0127] The one or more processors 1010 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0128] The one or more processors 1010 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the player window. The video image compositor may perform lens distortion correction on the one or more wide-angle cameras 1070, the one or more surround cameras 1074, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensor is preferably monitored by a neural network running on another instance of the enhanced SoC and configured to detect events in the cabin and respond accordingly.An in-cabin system can perform lip reading to activate cellular service and make a call, dictate emails, change the destination, activate or change the vehicle's infotainment system and settings, or enable voice-activated web browsing. Certain functions are available to the driver only when the vehicle is operating in autonomous mode and are disabled otherwise.

[0129] The video compositor can incorporate enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in a video, the noise reduction weights the spatial information accordingly, reducing the weight of information provided by neighboring frames. If a frame or section of a frame contains no motion, the temporal noise reduction performed by the video compositor can use information from the previous frame to reduce noise in the current frame.

[0130] The video image compositor may also be configured to perform stereo distortion correction on the input stereo lens images. The video image compositor may also be used for user interface design when the operating system desktop is in use and the one or more GPUs 1008 do not need to constantly render new surfaces. Even when the one or more GPUs 1008 are powered on and actively performing 3D rendering, the video image compositor may be used to offload the one or more GPUs 1008, thereby improving performance and responsiveness.

[0131] The one or more SoCs 1004 may also include a Mobile Industry Processor Interface (MIPI) serial camera interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that may be used for camera and related pixel input functions. The one or more SoCs 1004 may also include one or more input / output controllers, one or more of which may be software-controlled and used to receive I / O signals that are not associated with a specific role.

[0132] The one or more SoCs 1004 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The one or more SoCs 1004 may be used to process data from cameras (e.g., via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., one or more LIDAR sensors 1064, one or more RADAR sensors 1060, etc., which may be connected via Ethernet), data from bus 1002 (e.g., speed of vehicle 1000, steering wheel position, etc.), data from one or more GNSS sensors 1058 (e.g., connected via Ethernet or CAN bus).The one or more SoCs 1004 may further include dedicated high-performance mass storage controllers, which may include their own DMA engines and which may be used to offload routine data management tasks from the one or more CPUs 1006.

[0133] The one or more SoCs 1004 may be an end-to-end platform with a flexible architecture spanning automation levels 3-5, thereby providing a comprehensive functional safety architecture that supports and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The one or more SoCs 1004 may be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the one or more accelerators 1014 in combination with the one or more CPUs 1006, the one or more GPUs 1008, and the one or more data memories 1016 may form a fast, efficient platform for Level 3-5 autonomous vehicles.

[0134] The technology thus offers capabilities and functions that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using a high-level programming language, such as the C programming language, to execute a variety of processing algorithms on a wide variety of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, such as execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a prerequisite for in-vehicle ADAS applications and a requirement for practical Level 3-5 autonomous vehicles.

[0135] In contrast to conventional systems, the technology described herein enables the simultaneous and / or sequential execution of multiple neural networks and the combination of the results to enable Level 3-5 autonomous driving functionality by providing a CPU complex, a GPU complex, and a hardware acceleration cluster. For example, a CNN running on the DLA or dGPU (e.g., the one or more GPUs 1020) may include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing semantic understanding of the sign, and passing this semantic understanding to the path planning modules running on the CPU complex.

[0136] Another example is that multiple neural networks can run simultaneously, as required for driving at Level 3, 4, or 5. For example, a warning sign reading "Caution: Flashing lights indicate black ice," along with an electric light, can be interpreted independently or jointly by multiple neural networks. The sign itself can be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing lights indicate black ice" can be interpreted by a second deployed neural network, which informs the vehicle's path-planning software (preferably running on the CPU complex) that if flashing lights are detected, black ice is present.The turn signal can be identified across multiple images by a third neural network, which informs the vehicle's path planning software of the presence (or absence) of turn signals. All three neural networks can run simultaneously, e.g., within the DLA and / or on the one or more GPUs 1008.

[0137] In some examples, a facial recognition and vehicle owner identification CNN may use data from camera sensors to identify the presence of an authorized driver and / or owner of the vehicle 1000. The always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and to disable the vehicle in security mode when the owner exits the vehicle. In this way, the one or more SoCs 1004 provide security against theft and / or carjacking.

[0138] In another example, a CNN for detecting and identifying emergency vehicles may use data from microphones 1096 to detect and identify emergency vehicle sirens. Unlike conventional systems that use general classifiers to detect sirens and manually extract features, the one or more SoCs 1004 use the CNN to classify environmental and urban sounds, as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to detect the relative approach speed of the emergency vehicle (e.g., by using the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by one or more GNSS sensors 1058.For example, if the CNN is operating in Europe, it will attempt to detect European sirens, and if it is operating in the United States, the CNN will attempt to identify only North American sirens. Once an emergency vehicle is detected, a controller can be used to execute an emergency vehicle safety routine, slow the vehicle, pull over to the side of the road, park the vehicle, and / or idle the vehicle, using ultrasonic sensors 1062, until the one or more emergency vehicles pass by.

[0139] The vehicle may include one or more CPUs 1018 (e.g., one or more discrete CPUs or one or more dCPUs) that may be coupled to the one or more SoCs 1004 via a high-speed connection (e.g., PCIe). The CPUs 1018 may include, for example, an x86 processor. The CPUs 1018 may be used, for example, to perform a variety of functions, including reconciling potentially inconsistent results between ADAS sensors and the one or more SoCs 1004 and / or monitoring the status and health of the one or more controllers 1036 and / or the infotainment SoC 1030.

[0140] The vehicle 1000 may include one or more GPUs 1020 (e.g., one or more discrete GPUs or one or more dGPUs) that may be coupled to the one or more SoCs 1004 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The one or more GPUs 1020 may provide additional artificial intelligence capabilities, e.g., by executing redundant and / or distinct neural networks, and may be used to train and / or update neural networks based on inputs (e.g., sensor data) from sensors of the vehicle 1000.

[0141] The vehicle 1000 may further include the network interface 1024, which may include one or more wireless antennas 1026 (e.g., one or more wireless antennas for various communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 1024 may be used to enable a wireless connection over the internet to the cloud (e.g., to the one or more servers 1078 and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct connection between the two vehicles and / or an indirect connection may be established (e.g., via networks and the internet). Direct connections may be established via vehicle-to-vehicle communication.Vehicle-to-vehicle communication may provide vehicle 1000 with information about vehicles in the vicinity of vehicle 1000 (e.g., vehicles in front of, beside, and / or behind vehicle 1000). This functionality may be part of a cooperative adaptive cruise control function of vehicle 1000.

[0142] The network interface 1024 may include an SoC that provides modulation and demodulation functions and enables the one and more controllers 1036 to communicate over wireless networks. The network interface 1024 may include a radio frequency front end for upconverting from baseband to radio frequency and downconverting from radio frequency to baseband. The frequency conversions may be performed using known methods and / or super-heterodyne methods. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0143] The vehicle 1000 may further include one or more data stores 1028 that may be located off-chip (e.g., outside the SoCs 1004). The one or more data stores 1028 may include one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard drives, and / or other components and / or devices capable of storing at least one bit of data.

[0144] The vehicle 1000 may further include one or more GNSS sensors 1058. The one or more GNSS sensors 1058 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) assist in mapping, sensing, occupancy grid creation, and / or path planning. Any number of GNSS sensors 1058 may be used, including, for example, and without limitation, a GPS using a USB port with an Ethernet-to-serial (RS-232) bridge.

[0145] The vehicle 1000 may further include one or more RADAR sensors 1060. The one or more RADAR sensors 1060 may be used by the vehicle 1000 for long-range vehicle detection, even in darkness and / or poor weather conditions. The functional safety level of the RADAR may be ASIL B. The one or more RADAR sensors 1060 may utilize the CAN and / or bus 1002 (e.g., to transmit the data generated by the one or more RADAR sensors 1060) for control and access to object tracking data, with access to the raw data occurring over Ethernet in some examples. A variety of RADAR sensor types may be used. The one or more RADAR sensors 1060 may be suitable for, for example, front-, rear-, and side-facing RADAR, without limitation. In some examples, one or more Pulse Doppler RADAR sensors are used.

[0146] The one or more RADAR sensors 1060 may include various configurations, such as long range with a narrow field of view, short range with a wide field of view, short range side coverage, etc. In some examples, long range RADAR may be used for the adaptive cruise control function. Long range RADAR systems may provide a wide field of view realized through two or more independent scans, such as at a range of 250 m. The one or more RADAR sensors 1060 may assist in distinguishing between static and moving objects and may be used by ADAS systems for emergency braking and forward collision warning. Long range RADAR sensors may include a monostatic multi-modal RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface.In a six-antenna example, the middle four antennas can create a focused beam pattern designed to detect the surroundings of vehicle 1000 at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can expand the field of view so that vehicles entering or exiting the lane of vehicle 1000 can be quickly detected.

[0147] Medium-range radar systems, for example, can have a range of up to 1,060 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1,050 degrees (rear). Short-range radar systems can include radar sensors designed for installation at both ends of the rear bumper. When installed at both ends of the rear bumper, such a radar sensor system can generate two beams that continuously monitor the blind spot to the rear and to the side of the vehicle.

[0148] Short-range radar systems can be used in an ADAS system to monitor blind spots and / or assist with lane change.

[0149] The vehicle 1000 may also include one or more ultrasonic sensors 1062. The one or more ultrasonic sensors 1062, which may be mounted on the front, rear, and / or sides of the vehicle 1000, may be used for parking assistance and / or for creating and updating an occupancy grid. A plurality of ultrasonic sensors 1062 may be used, and different ultrasonic sensors 1062 may be used for different detection ranges (e.g., 2.5 m, 4 m). The one or more ultrasonic sensors 1062 may operate at functional safety levels of ASIL B.

[0150] The vehicle 1000 may include one or more LIDAR sensors 1064. The one or more LIDAR sensors 1064 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The one or more LIDAR sensors 1064 may be ASIL B functional safety level. In some examples, the vehicle 1000 may include multiple LIDAR sensors 1064 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to deliver data to a Gigabit Ethernet switch).

[0151] In some examples, the one or more LIDAR sensors 1064 may be capable of providing a list of objects and their distances for a 360-degree field of view. For example, commercially available LIDAR sensors 1064 may have an advertised range of approximately 1000 m, with an accuracy of 2 cm to 3 cm, and support for a 1000 Mbps Ethernet connection. In some examples, one or more non-prominent LIDAR sensors 1064 may be used. In such examples, the one or more LIDAR sensors 1064 may be implemented as a small device that may be embedded in the front, rear, sides, and / or corners of the vehicle 1000. In such examples, the one or more LIDAR sensors 1064 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m, even for objects with low reflectivity.The one or more front-mounted LIDAR sensors 1064 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0152] In some examples, LIDAR technologies, such as 3D flash LIDAR, may also be used. 3D flash LIDAR uses a laser flash as the transmitting source to illuminate the vehicle's surroundings up to approximately 200 m. A flash LIDAR unit contains a receptor that records the time of flight of the laser pulse and the reflected light at each pixel, which in turn corresponds to the distance between the vehicle and the objects. Flash LIDAR can enable highly accurate and distortion-free images of the surroundings to be created with each laser flash. In some examples, four flash LIDAR sensors may be deployed, one on each side of the vehicle. Available 3D flash LIDAR systems incorporate a solid-state 3D focal plane array LIDAR camera that contains no moving parts other than a fan (e.g., a non-scanning LIDAR device).The flash lidar device can use a 5-nanosecond pulse of a Class I (eye-safe) laser per frame and collect the reflected laser light as 3D range point clouds and co-registered intensity data. By using flash lidar, and because flash lidar is a solid-state device with no moving parts, the one or more lidar sensors 1064 can be less susceptible to motion blur, vibration, and / or shock.

[0153] The vehicle may also include one or more IMU sensors 1066. The one or more IMU sensors 1066 may, in some examples, be located at the center of the rear axle of the vehicle 1000. The one or more IMU sensors 1066 may, for example and without limitation, include one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other types of sensors. In some examples, such as in six-axis applications, the one or more IMU sensors 1066 may include accelerometers and gyroscopes, while in nine-axis applications, the one or more IMU sensors 1066 may include accelerometers, gyroscopes, and magnetometers.

[0154] In some embodiments, the one or more IMU sensors 1066 may be implemented as a miniaturized, high-performance GPS-aided inertial navigation system (GPS / INS) that combines microelectromechanical system (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and attitude. Thus, in some examples, the one or more IMU sensors 1066 may enable the vehicle 1000 to estimate heading without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS with the one or more IMU sensors 1066.In some examples, the one or more IMU sensors 1066 and the one or more GNSS sensors 1058 may be combined into a single integrated unit.

[0155] The vehicle may include one or more microphones 1096 mounted in and / or around the vehicle 1000. The one or more microphones 1096 may be used, among other things, to detect and identify emergency vehicles.

[0156] The vehicle may further include any number of camera types, including one or more stereo cameras 1068, one or more wide-angle cameras 1070, one or more infrared cameras 1072, one or more surround cameras 1074, one or more long-range and / or medium-range cameras 1098, and / or other camera types. The cameras may be used to capture image data around the entire periphery of the vehicle 1000. The types of cameras used depend on the embodiments and requirements for the vehicle 1000, and any combination of camera types may be used to provide the necessary coverage around the vehicle 1000. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or a different number of cameras.The cameras may support, by way of example and without limitation, Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the one or more cameras is referred to herein with reference to . Fig. 10A and Fig. 10B is described in more detail.

[0157] The vehicle 1000 may further include one or more vibration sensors 1042. The one or more vibration sensors 1042 may measure vibrations from components of the vehicle, such as the one or more axles. For example, changes in vibrations may indicate a change in the road surface. In another example, when two or more vibration sensors 1042 are used, the differences between the vibrations may be used to determine friction or slippage on the road surface (e.g., when the difference in vibration is between a driven axle and a free-spinning axle).

[0158] The vehicle 1000 may include an ADAS system 1038. The ADAS system 1038 may include an SoC in some examples. The ADAS system 1038 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (CWS), lane centering (LC), and / or other features and functions.

[0159] The ACC systems may use one or more RADAR sensors 1060, one or more LIDAR sensors 1064, and / or one or more cameras. The ACC systems may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle 1000 and automatically adjusts the vehicle speed to maintain a safe distance from preceding vehicles. Lateral ACC performs follow-through and advises the vehicle 1000 to change lanes if necessary. Lateral ACC is related to other ADAS applications, such as LCA and CWS.

[0160] The CACC utilizes information from other vehicles, which may be received via the network interface 1024 and / or the one or more wireless antennas 1026 from other vehicles over a wireless connection or indirectly via a network connection (e.g., over the Internet). Direct connections may be provided via a vehicle-to-vehicle (V2V) communication link, while indirect connections may be an infrastructure-to-vehicle (I2V) communication link. In general, the V2V communication concept provides information about the immediately preceding vehicles (e.g., vehicles immediately in front of and in the same lane as vehicle 1000), while the I2V communication concept provides information about traffic further ahead. CACC systems may include both I2V and V2V information sources.Given the information about the vehicles ahead of vehicle 1000, CACC can be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

[0161] FCW systems are designed to warn the driver of a hazard so they can take corrective action. FCW systems utilize a forward-facing camera and / or one or more radar sensors 1060 coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component. FCW systems can provide a warning, such as a sound, a visual warning, a vibration, and / or a rapid braking pulse.

[0162] AEB systems detect an impending forward collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. AEB systems can use one or more forward-facing cameras and / or one or more radar sensors 1060 coupled with a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision; if the driver does not take corrective action, the AEB system can automatically apply the brakes to prevent or at least mitigate the effects of the predicted collision. AEB systems can incorporate techniques such as dynamic brake support and / or impending crash braking.

[0163] LDW systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 1000 crosses lane markings. An LDW system will not activate if the driver indicates an intentional lane departure by activating a turn signal. LDW systems may utilize forward-facing cameras coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibrating component.

[0164] LKA systems are a variant of LDW systems. LKA systems provide steering inputs or braking to correct the vehicle 1000 when the vehicle 1000 begins to depart from its lane.

[0165] BSW systems detect and warn the driver of vehicles in the car's blind spot. BSW systems may provide a visual, audible, and / or tactile warning signal to indicate that merging into or changing lanes is unsafe. The system may provide an additional warning when the driver activates a turn signal. BSW systems may utilize one or more rear-facing cameras and / or radar sensors 1060 coupled with a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component.

[0166] RCTW systems can provide visual, audible, and / or tactile notification when an object outside the range of the rearview camera is detected when the vehicle 1000 is reversing. Some RCTW systems include AEB to ensure the vehicle brakes are applied to avoid a crash. RCTW systems can utilize one or more rear-facing RADAR sensors 1060 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component.

[0167] Conventional ADAS systems can produce false positives, which can be annoying and distracting for the driver, but are typically not catastrophic because ADAS systems warn the driver and give them the opportunity to decide whether a safety issue truly exists and act accordingly. However, in an autonomous vehicle 1000, in the event of conflicting results, the vehicle 1000 must decide for itself whether to consider the result of a primary computer or a secondary computer (e.g., a first controller 1036 or a second controller 1036). In some embodiments, the ADAS system 1038 may, for example, be a backup and / or secondary computer that provides perception information to a rationality module of the backup computer.The backup computer rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. The outputs of the ADAS system 1038 can be provided to a supervising MCU. If the outputs of the primary computer and the secondary computer conflict, the supervising MCU must determine how to resolve the conflict to ensure safe operation.

[0168] In some examples, the primary computer may be configured to provide the monitoring MCU with a confidence value indicating the primary computer's confidence in the chosen outcome. If the confidence value exceeds a threshold, the monitoring MCU may follow the primary computer's instruction regardless of whether the secondary computer provides a conflicting or inconsistent result. If the confidence value does not meet the threshold and the primary and secondary computers indicate different results (e.g., a conflict), the monitoring MCU may arbitrate between the computers to determine the appropriate outcome.

[0169] The monitoring MCU may be configured to run one or more neural networks trained and configured to determine the conditions under which the secondary computer triggers false alarms based on the output from the primary and secondary computers. This allows the one or more neural networks in the monitoring MCU to learn when the output of the secondary computer can be trusted and when it cannot. For example, if the secondary computer is a radar-based FCW system, a neural network in the monitoring MCU can learn when the FCW system identifies metallic objects that are not actually hazardous, such as a drain grate or manhole cover, which triggers an alarm.Similarly, if the secondary computer is a camera-based LDW system, a neural network in the monitoring MCU may learn to override the LDW system when cyclists or pedestrians are present and lane departure is indeed the safest maneuver. In embodiments including one or more neural networks running on the monitoring MCU, the monitoring MCU may include at least one DLA or GPU suitable for executing the one or more neural networks with associated memory. In preferred embodiments, the monitoring MCU may comprise and / or be included as a component of the one or more SoCs 1004.

[0170] In other examples, the ADAS system 1038 may include a secondary computer that executes the ADAS functionality according to classical computer vision rules. Thus, the secondary computer may use classical computer vision rules (if-then), and the presence of one or more neural networks in the supervising MCU may improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially against errors caused by software (or software-hardware interfaces).For example, if a software bug or error occurs in the software on the primary computer and the non-identical software code on the secondary computer produces the same overall result, the monitoring MCU can have greater confidence that the overall result is correct and the bug in the software or hardware on the primary computer does not cause a significant error.

[0171] In some examples, the output of ADAS system 1038 may be fed into the perception block of the primary computer and / or the dynamic driving task block of the primary computer. For example, if ADAS system 1038 displays a forward collision warning due to an object immediately in front of the vehicle, the perception block may use this information in identifying objects. In other examples, the secondary computer may have its own neural network trained to reduce the risk of false positives, as described herein.

[0172] The vehicle 1000 may further include the infotainment SoC 1030 (e.g., an in-vehicle infotainment (IVI) system). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 1030 may include a combination of hardware and software that can be used to provide the vehicle 1000 with audio (e.g., music, a personal digital assistant, navigation directions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, rear parking assist, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fuel level, oil level, door open / close, air filter information, etc.).The infotainment SoC 1030 may include, for example, radios, record players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free calling, a head-up display (HUD), an HMI display 1034, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. The infotainment SoC 1030 may further be used to provide information (e.g., visual and / or audible) to one or more users of the vehicle, such as information from the ADAS system 1038, autonomous driving information such as planned vehicle maneuvers, road layouts, environmental information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0173] The infotainment SoC 1030 may include GPU functionality. The infotainment SoC 1030 may communicate with other devices, systems, and / or components of the vehicle 1000 via the bus 1002 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1030 may be coupled to a supervisory MCU so that the GPU of the infotainment system can perform some self-driving functions if the one or more primary controllers 1036 (e.g., the primary and / or backup computers of the vehicle 1000) fail. In such an example, the infotainment SoC 1030 may place the vehicle 1000 into a chauffeur-to-safe-stop mode, as described herein.

[0174] The vehicle 1000 may further include an instrument cluster 1032 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1032 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 1032 may include a number of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag system (SRS) information, lighting controls, safety system controls, navigation information, etc. In some examples, information from the infotainment SoC 1030 and the instrument cluster 1032 may be displayed and / or shared. In other words, the instrument cluster 1032 may be included as part of the infotainment SoC 1030 or vice versa.

[0175] Fig. 10D is a system diagram for communication between the one or more cloud-based servers and the example autonomous vehicle 1000 of Fig. 10A, according to some embodiments of the present disclosure. The system 1076 may include the one or more servers 1078, the one or more networks 1090, and the vehicles, including the vehicle 1000. The server(s) 1078 may include a plurality of GPUs 1084(A)-1084(H) (collectively referred to herein as GPUs 1084), PCIe switches 1082(A)-1082(H) (collectively referred to herein as PCIe switches 1082), and / or CPUs 1080(A)-1080(B) (collectively referred to herein as CPUs 1080). The GPUs 1084, the CPUs 1080, and the PCIe switches may be interconnected using high-speed interconnects, such as, without limitation, the NVIDIA-developed NVLink interfaces 1088 and / or PCIe interconnects 1086. In some examples, the GPUs 1084 are interconnected via NVLink and / or NVSwitch SoC, and the GPUs 1084 and the PCIe switches 1082 are interconnected via PCIe interconnects.Although eight GPUs 1084, two CPUs 1080, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 1078 may include any number of GPUs 1084, CPUs 1080, and / or PCIe switches. For example, the one or more servers 1078 may each include eight, sixteen, thirty-two, and / or more GPUs 1084.

[0176] The one or more servers 1078 may receive, via the one or more networks 1090 and from the vehicles, image data representative of images depicting unexpected or changed road conditions, such as recently commenced roadwork. The one or more servers 1078 may transmit, via the one or more networks 1090 and to the vehicles, neural networks 1092, updated neural networks 1092, and / or map information 1094 containing information about traffic and road conditions. The updates to the map information 1094 may include updates to the HD map 1022, such as information about construction, potholes, detours, flooding, and / or other obstacles.In some examples, the neural networks 1092, the updated neural networks 1092, and / or the map information 1094 may result from new training and / or experience represented in the data received from any number of vehicles in the environment and / or based on training performed in a data center (e.g., using the one or more servers 1078 and / or other servers).

[0177] The one or more servers 1078 may be used to train machine learning models (e.g., neural networks) based on training data. The training data may be generated by the vehicles and / or in a simulation (e.g., using a game engine). In some examples, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not tagged and / or preprocessed (e.g., if the neural network does not require supervised learning).Training may be performed using one or more classes of machine learning techniques, including, without limitation, supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including replacement dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning models are trained, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles via the one or more networks 1090) and / or the machine learning models may be used by the one or more servers 1078 to remotely monitor the vehicles.

[0178] In some examples, the one or more servers 1078 may receive data from the vehicles and apply the data to real-time, real-time neural networks for intelligent inference. The one or more servers 1078 may include deep learning supercomputers and / or dedicated AI computers powered by GPUs 1084, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the one or more servers 1078 may include a deep learning infrastructure using only CPU-powered data centers.

[0179] The deep learning infrastructure of the one or more servers 1078 may be capable of performing rapid inference in real time and may utilize this capability to evaluate and verify the state of the processors, software, and / or associated hardware in the vehicle 1000. For example, the deep learning infrastructure may receive periodic updates from the vehicle 1000, such as a sequence of images and / or objects that the vehicle 1000 has located in that sequence of images (e.g., via computer vision and / or other machine learning object classification techniques).The deep learning infrastructure may run its own neural network to identify the objects and compare them to the objects identified by the vehicle 1000, and if the results do not match and the infrastructure concludes that the AI ​​in the vehicle 1000 is not functioning properly, the one or more servers 1078 may send a signal to the vehicle 1000 instructing a fail-safe computer of the vehicle 1000 to take control, notify passengers, and perform a safe parking maneuver.

[0180] For inferencing, the one or more servers 1078 may include GPUs 1084 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-driven servers and inference accelerators may enable real-time responsiveness. In other examples, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors may be used for inferencing. EXAMPLE CALCULATION DEVICE

[0181] Fig. 11 is a block diagram of an example computing device 1100 suitable for use in implementing some embodiments of the present disclosure. Computing device 1100 may include an interconnect system 1102 that directly or indirectly couples the following devices: memory 1104, one or more central processing units (CPUs) 1106, one or more graphics processing units (GPUs) 1108, a communications interface 1110, input / output (I / O) ports 1112, input / output components 1114, a power supply 1116, one or more presentation components 1118 (e.g., display(s)), and one or more logic units 1120. In at least one embodiment, the one or more computing devices 1100 may include one or more virtual machines (VMs), and / or each of the components thereof may include virtual components (e.g., virtual hardware components).As non-limiting examples, one or more of the GPUs 1108 may include one or more vGPUs, one or more of the CPUs 1106 may include one or more vCPUs, and / or one or more of the logic units 1120 may include one or more virtual logic units. Thus, a computing device 1100 may include discrete components (e.g., a full GPU associated with the computing device 1100), virtual components (e.g., a portion of a GPU associated with the computing device 1100), or a combination thereof.

[0182] Although the different blocks of Fig. 11 are illustrated as being connected to wires via the interconnect system 1102, this is not intended as a limitation and is for clarity only. For example, in some embodiments, a presentation component 1118, such as a display device, may be considered an I / O component 1114 (e.g., if the display is a touchscreen). As another example, the CPUs 1106 and / or GPUs 1108 may include memory (e.g., the memory 1104 may represent a storage device in addition to the memory of the GPUs 1108, the CPUs 1106, and / or other components). In other words, the computing device of Fig. 11 is merely illustrative. No distinction is made between categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, since all devices within the scope of the computing device are covered by Fig. 11 may be considered.

[0183] The interconnect system 1102 may represent one or more connections or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 1102 may include one or more bus or connection types, such as an Industry Standard Architecture (ISA) bus, an Extended ISA bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, and / or another type of bus or connection. In some embodiments, there are direct connections between components. For example, the CPU 1106 may be directly connected to the memory 1104. Further, the CPU 1106 may be directly connected to the GPU 1108.For a direct or point-to-point connection between components, interconnect system 1102 may include a PCIe link to establish the connection. In these examples, a PCI bus need not be included in computing device 1100.

[0184] Memory 1104 may include a variety of computer-readable media. The computer-readable media may be any available media accessible by the computing device 1100. The computer-readable media may include both volatile and non-volatile media, as well as removable and non-removable media. By way of example and without limitation, the computer-readable media may include computer storage media and communication media.

[0185] The computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, the memory 1104 may store computer-readable instructions (e.g., representing one or more programs) and / or one or more program elements, such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 1100.As used herein, computer storage media does not per se include signals.

[0186] The computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and may include any media for conveying information. The term "modulated data signal" may refer to a signal having one or more of its characteristics adjusted or altered to encode information in the signal. The computer storage media may include, for example, and is not limited to, wired media, such as a wired network or a direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of the above should also be within the scope of computer-readable media.

[0187] The one or more CPUs 1106 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the methods and / or processes described herein. The one or more CPUs 1106 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of concurrently executing a plurality of software threads. The one or more CPUs 1106 may include any type of processor and may include different types of processors depending on the type of computing device 1100 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).Depending on the type of computing device 1100, the processor may be, for example, an Advanced RISC Machines (ARM) processor implemented with Reduced Instruction Set Computing (RISC) or an x86 processor implemented with Complex Instruction Set Computing (CISC). Computing device 1100 may include one or more CPUs 1106, in addition to one or more microprocessors or additional coprocessors, such as math coprocessors.

[0188] In addition or alternatively to the one or more CPUs 1106, the one or more GPUs 1108 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 1108 may be an integrated GPU (e.g., with one or more of the CPUs 1106 and / or one or more of the GPUs 1108 may be a discrete GPU. In embodiments, one or more of the GPUs 1108 may be a co-processor of one or more of the CPUs 1106. The one or more GPUs 1108 may be used by the computing device 1100 to render graphics (e.g., 3D graphics) or to perform general-purpose computations. The one or more GPUs 1108 may be used, for example, for general-purpose computing on GPUs (GPGPU).The one or more GPUs 1108 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The one or more GPUs 1108 may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the one or more CPUs 1106 received via a host interface). The one or more GPUs 1108 may include graphics memory, such as display memory, for storing pixel data or other suitable data, such as GPGPU data. The display memory may be included as part of the memory 1104. The one or more GPUs 1108 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or connect the GPUs via a switch (e.g., using NVSwitch).When combined, each GPU can generate 1108 pixel data or GPGPU data for different sections of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can contain its own memory or share memory with other GPUs.

[0189] In addition to or alternatively to the one or more CPUs 1106 and / or the one or more GPUs 1108, the one or more logic units 1120 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1100 to perform one or more of the methods and / or processes described herein. In embodiments, the one or more CPUs 1106, the GPUs 1108, and / or the one or more logic units 1120 may discretely or jointly execute any combination of the methods, processes, and / or portions thereof. One or more of the logic units 1120 may be part of and / or integrated with one or more of the CPUs 1106 and / or one or more of the GPUs 1108, and / or one or more of the logic units 1120 may be discrete components or otherwise external to the CPUs 1106 and / or the GPUs 1108.In embodiments, one or more of the logic units 1120 may be a co-processor of one or more of the CPUs 1106 and / or one or more of the GPUs 1108.

[0190] Examples of the one or more logic units 1120 include one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (Application-Specific Integrated Circuits, ASICs), Floating Point Units (FPUs),Input / output (I / O) elements, peripheral component interconnect (PCI) or PCI Express (PCIe) elements, and / or similar.

[0191] The communication interface 1110 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1100 to communicate with other computers over an electronic network, including wired and / or wireless communication. The communication interface 1110 may include components and functions that enable communication over a variety of networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.In one or more embodiments, the one or more logic units 1120 and / or the communication interface 1110 may include one or more data processing units (DPUs) to transfer data received over a network and / or via the interconnect system 1102 directly to one or more GPUs 1108 (e.g., a memory thereof).

[0192] The I / O ports 1112 may enable the computing device 1100 to be logically coupled to other devices, including the I / O components 1114, the one or more presentation components 1118, and / or other components, some of which may be built into (e.g., integrated) the computing device 1100. Illustrative I / O components 1114 include a microphone, a mouse, a keyboard, a joystick, a gamepad, a game controller, a satellite dish, a scanner, a printer, a wireless device, etc. The I / O components 1114 may provide a natural user interface (NUI) that processes air gestures, speech, or other physiological inputs generated by a user. In some cases, the inputs may be communicated to a suitable network element for further processing.An NUI may implement any combination of speech capture, stylus capture, facial capture, biometric capture, both on-screen and off-screen gesture capture, air gestures, head and eye tracking, and touch capture (as further described below) associated with a display of the computing device 1100. The computing device 1100 may include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture capture and recognition. Additionally, the computing device 1100 may include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable the capture of motion. In some examples, the output of the accelerometers or gyroscopes from the computing device 1100 may be used to render immersive augmented reality or virtual reality.

[0193] Power supply 1116 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 1116 may supply power to computing device 1100 to enable operation of the components of computing device 1100.

[0194] The one or more presentation components 1118 may include a display (e.g., a monitor, a touchscreen, a television monitor, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The one or more presentation components 1118 may receive data from other components (e.g., the one or more GPUs 1108, the one or more CPUs 1106, DPUs, etc.) and output the data (e.g., as an image, video, audio, etc.). EXEMPLARY DATA CENTER

[0195] Fig. 12 illustrates an example data center 1200 that may be used in at least one embodiment of the present disclosure. Data center 1200 may include a data center infrastructure layer 1210, a framework layer 1220, a software layer 1230, and / or an application layer 1240.

[0196] As in Fig. 12, the data center infrastructure layer 1210 may include a resource orchestrator 1212, clustered computing resources 1214, and node computing resources (“node CRs”) 1216(1)-1216(N), where “N” represents any positive integer. In at least one embodiment, the node CRs 1216(1)-1216(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc.In some embodiments, one or more Node CRs among Node CRs 1216(1)-1216(N) may correspond to a server having one or more of the computing resources mentioned above. Furthermore, in some embodiments, Node CRs 1216(1)-1216(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or the like, and / or one or more of Node CRs 1216(1)-1216(N) may correspond to a virtual machine (VM).

[0197] In at least one embodiment, the grouped computing resources 1214 may include separate groupings of node CRs 1216 housed in one or more racks (not shown) or in many racks in data centers in different geographic locations (also not shown). Separate groupings of node CRs 1216 within grouped computing resources 1214 may include grouped computing, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs 1216, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads.The one or more racks may also contain any number of power modules, cooling modules, and / or network switches in any combination.

[0198] Resource orchestrator 1212 may configure or otherwise control one or more node CRs 1216(1)-1216(N) and / or clustered computing resources 1214. In at least one embodiment, resource orchestrator 1212 may include an entity for managing the software design infrastructure (SDI) for data center 1200. Resource orchestrator 1212 may include hardware, software, or a combination thereof.

[0199] In at least one embodiment, as in Fig. 12, the framework layer 1220 may include a job scheduler 1233, a configuration manager 1234, a resource manager 1236, and / or a distributed file system 1238. The framework layer 1220 may include a framework that supports the software 1232 of the software layer 1230 and / or one or more applications 1242 of the application layer 1240. The software 1232 or the one or more applications 1242 may each include web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1220 may be some type of free and open source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter "Spark"), which may utilize a distributed file system 1238 for processing large amounts of data (e.g., "Big Data").In at least one embodiment, job scheduler 1233 may include a Spark driver to facilitate scheduling workloads supported by various layers of data center 1200. Configuration manager 1234 may be capable of configuring various layers, such as software layer 1230 and framework layer 1220, which includes Spark and distributed file system 1238, to support processing large amounts of data. Resource manager 1236 may be capable of managing clustered or grouped computing resources allocated or assigned to support distributed file system 1238 and job scheduler 1233. In at least one embodiment, the clustered or grouped computing resources may include clustered computing resource 1214 at infrastructure layer 1210 of the data center.The resource manager 1236 may coordinate with the resource orchestrator 1212 to manage these allocated or assigned computing resources.

[0200] In at least one embodiment, the software 1232 included in software layer 1230 may include software used by at least portions of node CRs 1216(1)-1216(N), clustered computing resources 1214, and / or distributed file system 1238 of framework layer 1220. One or more types of software may include, but are not limited to, web page searching software, email virus scanning software, database software, and streaming video content software.

[0201] In at least one embodiment, the applications 1242 included in the application layer 1240 may include one or more types of applications used by at least portions of the node CRs 1216(1)-1216(N), the clustered computing resources 1214, and / or the distributed file system 1238 of the framework layer 1220. One or more types of applications may include, but are not limited to, any number of genomic applications, cognitive computation, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in connection with one or more embodiments.

[0202] In at least one embodiment, one of a configuration manager 1234, resource manager 1236, and resource orchestrator 1212 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically feasible manner. Self-modifying actions may relieve a data center operator of data center 1200 from making potentially poor configuration decisions and potentially avoid underutilized and / or poorly performing sections of a data center.

[0203] Data center 1200 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, one or more machine learning models may be trained by calculating weighting parameters according to a neural network architecture using software and / or computing resources described above with reference to data center 1200.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with reference to data center 1200 using weighting parameters calculated by one or more training techniques, such as, but not limited to, those described herein.

[0204] In at least one embodiment, the data center 1200 may utilize CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inferencing using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as a service to enable users to train or infer information, such as image capture, speech capture, or other artificial intelligence services. EXAMPLE NETWORK ENVIRONMENTS

[0205] Network environments suitable for implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be based on one or more instances of the one or more computing devices 1100 of Fig. 11 - e.g., each device may include similar components, features, and / or functionality of the one or more computing devices 1100. Furthermore, if backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be included as part of a data center 1200, an example of which is described herein with reference to Fig. 12 is described in more detail.

[0206] The components of a network environment can communicate with each other over one or more networks, which can be wired, wireless, or both. The network can contain multiple networks or a network of networks. For example, the network can contain one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or a public switched telephone network (PSTN), and / or one or more private networks. If the network contains a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0207] Compatible network environments may include one or more peer-to-peer network environments—in which case, a server may not be included in a network environment—and one or more client-server network environments—in which case, one or more servers may be included in a network environment. In peer-to-peer network environments, the functionality described herein may be implemented with reference to one or more servers on any number of client devices.

[0208] In at least one embodiment, a network environment may include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. A framework layer may include a framework for supporting software of a software layer and / or one or more applications of an application layer. The software or the one or more applications may each include web-based service software or applications. In embodiments, one or more of the client devices may use the web-based service software or applications (e.g.,by accessing the service software and / or applications through one or more application programming interfaces (APIs). The framework layer may be some type of free and open-source software web application framework, e.g., using, but not limited to, a distributed file system for processing large amounts of data (e.g., "Big Data").

[0209] A cloud-based network environment may provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Each of these various functions may be distributed across multiple locations of central or core servers (e.g., one or more data centers that may be located across a state, region, country, globe, etc.). When a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers may offload at least some functionality to the one or more edge servers. A cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0210] The one or more client devices may include at least some of the components, features, and functions of the one or more described herein with respect to Fig.11 described exemplary computing devices 1100.By way of example, and not limitation, a client device may be embodied as a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or global positioning device, a video player, a video camera, a surveillance device or surveillance system, a vehicle, a boat, a flying boat, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or gaming system, an entertainment system, a vehicle computing system, an embedded system controller, a remote control, an appliance, a consumer electronics device, a workstation, an edge device, any combination of these described devices, or any other suitable device.

[0211] The disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., and refer to code that performs specific tasks or implements specific abstract data types. The disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc.The disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are interconnected via a network for communication.

[0212] As used herein, any reference to "and / or" in reference to two or more elements should be interpreted to mean only one element or a combination of elements. For example, "Element A, Element B, and / or Element C" may include only Element A, only Element B, only Element C, Element A and Element B, Element A and Element C, Element B and Element C, or Elements A, B, and C. Furthermore, "at least one of Element A or Element B" may include at least one of Element A, at least one of Element B, or at least one of Element A and at least one of Element B. Further, "at least one of Element A and Element B" may include at least one of Element A, at least one of Element B, or at least one of Element A and at least one of Element B.

[0213] The subject matter of the present disclosure is specifically described herein to satisfy legal requirements. However, the description itself is not intended to limit the scope of the present disclosure. Rather, the inventors have contemplated that the claimed subject matter may be embodied in other ways to include various steps or combinations of steps similar to those described herein, in conjunction with other present or future technologies. Although the terms "step" and / or "block" may be used herein to refer to various elements of the methods employed, the terms should not be interpreted to imply any particular ordering among or between the various steps disclosed herein, unless and except that the order of the individual steps is expressly described.

[0214] The disclosure of this application also contains the following numbered clauses:

[0215] Clause 1 A method comprising: obtaining data associated with a traffic feature located within an environment during one or more trips; determining, based at least on map data, one or more candidate lines associated with the traffic feature; determining, based at least on the one or more trips, that a candidate line of the one or more candidate lines includes a wait line for the traffic feature; and updating the map data to indicate that the candidate line includes the wait line for the traffic feature.

[0216] Clause 2 The method of Clause 1, further comprising: determining one or more rules associated with the traffic feature, and determining, based at least on the one or more rules, one or more numerical values ​​associated with the one or more trips, wherein determining that the candidate line includes the waiting line for the traffic feature is further based at least on the one or more numerical values.

[0217] Clause 3 The method of Clause 2, wherein determining the one or more numerical values ​​associated with the one or more trips comprises one or more of: determining a first numerical value associated with a first trip based at least on a first machine associated with the first trip following the one or more rules; or determining a second numerical value associated with a second trip based at least on a second machine associated with the second trip not following the one or more rules, wherein the second numerical value is less than the first numerical value.

[0218] Clause 4 The method of any of clauses 2 or 3, wherein: the one or more trips include a plurality of trips associated with the traffic feature; the one or more numerical values ​​include a plurality of numerical values ​​associated with the plurality of trips; the method further comprises determining, based at least on removing a first portion of the plurality of trips associated with a portion of the plurality of numerical values ​​that are less than a threshold, a second portion of the plurality of trips; and determining that the candidate line includes the waiting line is based at least on the second portion of the plurality of trips.

[0219] Clause 5 The method of any preceding clause, further comprising: determining that the one or more trips are associated with a lane of travel to one or more lanes located within the environment; and determining a group of trips based at least on merging the one or more trips associated with the lane of travel, wherein determining that the candidate line includes the waiting line for the traffic feature is based at least on the group of trips.

[0220] Clause 6 The method of any preceding clause, further comprising: determining, based at least on the map data, that the one or more candidate lines are associated with a same lane as the traffic feature, wherein determining the one or more candidate lines associated with the traffic feature is based at least on the one or more candidate lines being associated with the same lane as the traffic feature.

[0221] Clause 7 The method of any preceding clause, further comprising: determining, based at least on the map data, that the one or more candidate lines are within a threshold distance from the traffic feature, wherein determining the one or more candidate lines associated with the traffic feature is based at least on the one or more candidate lines being within the threshold distance from the traffic feature.

[0222] Clause 8 A method according to any preceding clause, wherein: the traffic feature comprises one or more of: a traffic signal; a stop sign; a crosswalk signal; a zebra crossing sign; a level crossing signal; a railway crossing sign; a yield sign; or a stop light; and the one or more candidate lines comprise one or more of: a stop line; a crosswalk line; an intersection entry line; an intersection exit line; a railway crossing line; or a yield line.

[0223] Clause 9 A method according to any preceding clause, further comprising sending the map data after the update to one or more machines navigating within the environment, the map data causing the one or more machines to stop at the waiting line when approaching the traffic feature.

[0224] Clause 10 A system comprising: one or more processors for: determining, based at least on map data, a traffic feature located within an environment; obtaining data representing one or more trips associated with the traffic feature; determining, based at least on the one or more trips, a wait line associated with the traffic feature; and updating the map data to indicate that the wait line is associated with the traffic feature.

[0225] Clause 11 The system of Clause 10, wherein the one or more processors are further to: determine, based at least on the map data, one or more candidate lines associated with the traffic feature, wherein determining the waiting line comprises determining, based at least on the one or more trips, that a candidate line of the one or more candidate lines includes the waiting line associated with the traffic feature.

[0226] Clause 12 The system of Clause 11, wherein the one or more processors are further to: determine, based at least on the map data, that the one or more candidate lines are within a threshold distance from the traffic feature, wherein the determination of the one or more candidate lines associated with the traffic feature is based at least on the one or more candidate lines being within the threshold distance from the traffic feature.

[0227] Clause 13 The system of any of clauses 10-12, wherein the one or more processors are further to: determine, based on at least the one or more trips, one or more locations within the environment at which one or more machines associated with the one or more trips will stop when approaching the traffic feature, wherein determining the wait line comprises determining the wait line associated with the traffic feature based on at least the one or more locations.

[0228] Clause 14 The system of any of clauses 10-13, wherein the one or more processors are further to: determine one or more rules associated with the traffic feature; and determine, based at least on the one or more rules, one or more numerical values ​​associated with the one or more trips, wherein the determination of the wait line associated with the traffic feature is further based at least on the one or more numerical values.

[0229] Clause 15 The system of paragraph N, wherein determining the one or more numerical values ​​associated with the one or more trips comprises one or more of: determining a first numerical value associated with a first trip based at least on a first machine associated with the first trip following the one or more rules; or determining a second numerical value associated with a second trip based at least on a second machine associated with the second trip not following the one or more rules, wherein the second numerical value is less than the first numerical value.

[0230] Clause 16 A system according to clause 14 or 15, wherein: the one or more trips include a plurality of trips associated with the traffic characteristic; the one or more numerical values ​​include a plurality of numerical values ​​associated with the plurality of trips; the one or more processors are further to determine a second portion of the plurality of trips based at least on removing a first portion of the plurality of trips associated with a portion of the plurality of numerical values ​​that are less than a threshold; and determining the wait line associated with the traffic characteristic is based at least on the second portion of the plurality of trips.

[0231] Clause 17 The system of any of clauses 10-16, wherein the one or more processors are further to: determine that the one or more trips are associated with a lane of travel to one or more lanes located within the environment; and determine a group of trips based at least on merging the one or more trips associated with the lane of travel, wherein the determination of the wait line associated with the traffic feature is based at least on the group of trips.

[0232] Clause 18: A system according to any of Clauses 10-17, wherein the system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs);a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0233] Clause 19 One or more processors comprising: processing circuitry for updating map data to indicate that a candidate line of one or more candidate lines associated with a traffic feature located within an environment includes a wait time line associated with the traffic feature, the candidate line being selected as including the wait time line based on at least one or more trips associated with one or more machines navigating within the environment and through an intersection associated with the traffic feature.

[0234] Clause 20 The one or more processors of Clause 19, wherein the one or more processors comprise at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using one or more large language models (LLMs);a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is at least partially implemented using cloud computing resources.

[0235] Any, some, or all features of one aspect of the disclosure may be applied in any suitable combination or subcombination to any other suitable aspects of the disclosure. In particular, apparatus aspects may be applied to method aspects, and vice versa. It is also understood that certain combinations of the various features described and defined in each aspect or embodiment of the disclosure may be implemented and / or provided and / or used independently of one another.

[0236] The various features described in the description as optional, such as using "may" or "can", may be combined into a single embodiment and / or any combination of the features may be combined to form various embodiments based on the combination of these various optional features.

[0237] It is to be understood that aspects and embodiments described above are purely exemplary and that modifications of details may be made within the scope of the claims.

[0238] Each device, method, and feature disclosed in the description, and (where appropriate) the claims and drawings may be provided independently or in any suitable combination.

[0239] Reference signs appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 16 / 101,232

[0116] Cited non-patent literature

[0000] J3016-201806

[0074] J3016-201609

[0074]

Claims

[1] Method comprising: Obtaining data associated with a traffic feature located within an environment during one or more trips; Determining, based at least on map data, one or more candidate lines associated with the traffic feature; Determining, based on at least the one or more trips, that a candidate line of the one or more candidate lines includes a waiting line for the traffic feature; and Update the map data to indicate that the candidate line contains the waiting line for the traffic feature. [2] The method of claim 1, further comprising: Determining one or more rules associated with the traffic feature; and Determining, based on at least the one or more rules, one or more numerical values ​​associated with the one or more trips, wherein determining that the candidate line includes the waiting line for the traffic feature is further based on at least the one or more numerical values [3] The method of claim 2, wherein determining the one or more numerical values ​​associated with the one or more trips comprises one or more of the following: Determining a first numerical value associated with a first trip based at least on a first machine associated with the first trip following the one or more rules; or Determining a second numerical value associated with a second trip based at least on a second machine associated with the second trip not following the one or more rules, wherein the second numerical value is less than the first numerical value. [4] A method according to any one of claims 2 or 3, wherein: the one or more journeys contain a plurality of journeys assigned to the traffic feature; the one or more numerical values ​​include a plurality of numerical values ​​associated with the plurality of journeys; the method further comprises determining, based at least on removing a first portion of the plurality of trips associated with a portion of the plurality of numerical values ​​that are less than a threshold, a second portion of the plurality of trips; and determining that the candidate line includes the waiting line is based on at least the second part of the plurality of trips. [5] Method according to one of the preceding claims, further comprising: Determining that the one or more lane trips are associated with one or more lanes located within the environment; and Determining a group of journeys based at least on merging the one or more journeys associated with the lane, wherein determining that the candidate line contains the waiting line for the traffic feature is based at least on the set of trips. [6] Method according to one of the preceding claims, further comprising: Determining, based at least on the map data, that the one or more candidate lines are assigned to the same lane as the traffic feature, wherein determining the one or more candidate lines associated with the traffic feature is based at least on the one or more candidate lines being associated with the same lane as the traffic feature. [7] Method according to one of the preceding claims, further comprising: Determining, based at least on the map data, that the one or more candidate lines are within a threshold distance from the traffic feature, wherein determining the one or more candidate lines associated with the traffic feature is based at least on the one or more candidate lines being within the threshold distance to the traffic feature. [8] Method according to one of the preceding claims, wherein: the traffic characteristic includes one or more of the following: a traffic light; a stop sign; a zebra crossing traffic light; a zebra crossing sign; a railway crossing traffic light; a railway crossing sign; a give-way sign; or a stop light; and the one or more candidate lines include one or more of the following: a stop line; a zebra crossing line; an intersection entry line; an intersection exit line; a level crossing line; or a give-way line. [9] The method of any preceding claim, further comprising sending the map data after the update to one or more machines navigating within the environment, the map data causing the one or more machines to stop at the waiting line when approaching the traffic feature. [10] System comprising: one or more processors, for: Determining, based at least on map data, a traffic feature located within an environment; Obtaining data representing one or more journeys associated with the traffic feature; Determining, based on at least the one or more trips, a waiting line associated with the traffic feature; and Update the map data to indicate that the waiting line is associated with the traffic feature. [11] The system of claim 10, wherein the one or more processors are further configured to: Determine, based at least on the map data, one or more candidate lines associated with the traffic feature, wherein determining the waiting line comprises determining, based on at least the one or more trips, that a candidate line of the one or more candidate lines includes the waiting line associated with the traffic feature. [12] The system of claim 11, wherein the one or more processors are further configured to: Determining, based at least on the map data, that the one or more candidate lines are within a threshold distance from the traffic feature, wherein the determination of the one or more candidate lines associated with the traffic feature is based at least on the one or more candidate lines being within the threshold distance to the traffic feature. [13] The system of any of claims 10-12, wherein the one or more processors further serve to: Determining, based on at least the one or more trips, one or more locations within the environment at which one or more machines associated with the one or more trips will stop when approaching the traffic feature, wherein determining the waiting line comprises determining the waiting line associated with the traffic feature based on at least the one or more locations. [14] The system of any of claims 10-13, wherein the one or more processors further serve to: Determining one or more rules associated with the traffic feature; and Determining, based on at least the one or more rules, one or more numerical values ​​associated with the one or more trips, wherein the determination of the waiting line associated with the traffic feature is further based on at least the one or more numerical values. [15] The system of claim 14, wherein determining the one or more numerical values ​​associated with the one or more trips comprises one or more of the following: Determining a first numerical value associated with a first trip based at least on a first machine associated with the first trip following the one or more rules; or Determining a second numerical value associated with a second trip based at least on a second machine associated with the second trip not following the one or more rules, wherein the second numerical value is less than the first numerical value. [16] System according to one of claims 14 or 15, wherein: the one or more journeys contain a plurality of journeys assigned to the traffic feature; the one or more numerical values ​​include a plurality of numerical values ​​associated with the plurality of journeys; the one or more processors are further operable to determine a second portion of the plurality of trips based at least on removing a first portion of the plurality of trips associated with a portion of the plurality of numerical values ​​that are less than a threshold; and the determination of the waiting line associated with the traffic feature is based on at least the second part of the plurality of journeys. [17] The system of any of claims 10-16, wherein the one or more processors further serve to: Determining that the one or more lane trips are associated with one or more lanes located within the environment; and Determining a group of journeys based at least on merging the one or more journeys associated with the lane, where the determination of the waiting line associated with the traffic feature is based at least on the group of trips. [18] A system according to any one of claims 10-17, wherein the system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more operations using generative AI; a system for performing operations using one or more large language models (LLMs); a system for performing one or more operations using conversational AI; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources. [19] One or more processors comprising: Processing circuitry for updating map data to indicate that a candidate line of one or more candidate lines associated with a traffic feature located within an environment includes a wait time line associated with the traffic feature, the candidate line being selected as including the wait time line based on at least one or more trips associated with one or more machines navigating within the environment and through an intersection associated with the traffic feature. [20] The one or more processors of claim 19, wherein the one or more processors comprise at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more operations using generative AI; a system for performing operations using one or more large language models (LLMs); a system for performing one or more operations using conversational AI; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources.

Images