Aligning navigation road graph and perception lane graph

Abstract

In various examples, embodiments are directed to navigation road and perception lane matching for autonomous and semi-autonomous systems and applications. In this regard, lane-road matching is performed using geometric similarity to generate effective lane-road assignments used in, among other things, lane planning and decision making. In some embodiments, road data representing at least one road segment and lane data representing a lane associated with an ego machine location are received. Thereafter, a determination is made that one or more consecutive road segments match a lane based at least on a geometric similarity between the lane and the one or more consecutive road segments.A representation of the lane assigned to the one or more consecutive road segments is generated based at least on determining that the one or more consecutive road segments have been matched to the lane.

Description

BACKGROUND

[0001] Developing a system to safely drive a vehicle autonomously without supervision or semi-autonomously with limited supervision is an extremely challenging task. For example, some designs envision an autonomous vehicle or other ego-machine being able to act as the functional equivalent of an attentive driver, relying on a perception and action system with an incredible ability to detect and react to dynamic and static obstacles in a complex environment to avoid other objects or structures along the vehicle's path. In developing such a system, lane graphs can be used to perform tasks such as localization (e.g., positioning an ego-machine on a map), path planning (e.g., determining candidate routes for an ego-machine), object detection and tracking (e.g.,to perform object-in-path analysis (OIPA), etc.) and / or facilitate decision-making (e.g., determining an optimal route based on candidate routes, vehicle condition, and the environment). For example, a lane graph can be used in lane planning to make decisions about lane changes, turning maneuvers, and lane staying. Lane planning in the context of autonomous vehicles is useful for many purposes, including, but not limited to, following planned routes and making turns to navigate in an organized manner, performing efficient maneuvers, and facilitating lane staying assistance. Accordingly, accurate lane graphs are useful for effectively facilitating localization, lane planning, and / or decision-making.

[0002] However, in some circumstances, a lane graph may lack adequate data reflected in the lane graph, thereby impairing the effectiveness of the lane graph (e.g., for use in performing lane planning). For example, a lane graph may contain inaccurate or incomplete lane widths, markings, or other attributes, which may lead to suboptimal lane planning and navigation decision-making. Accordingly, planning and decision-making using such lane graphs may result in inaccurate or inefficient information, limiting the ability of an autonomous vehicle to effectively navigate through an environment. SUMMARY

[0003] The invention is defined by the claims. To illustrate the invention, aspects and embodiments are described herein, which may or may not fall within the scope of the claims.

[0004] Various embodiments are disclosed directed to navigation road and perception lane matching for autonomous and semi-autonomous systems and applications. In this regard, lane-road matching is performed using geometric similarity to generate effective lane-road assignments used in, among other things, lane planning and decision making. In some embodiments, road data representing at least one road segment and lane data representing a lane associated with a location of an ego machine are received. Thereafter, a determination is made that one or more consecutive road segments match a lane based at least on a geometric similarity between the lane and the one or more consecutive road segments.A representation of the lane assigned to the one or more consecutive road segments is generated based at least on determining that the one or more consecutive road segments have been matched to the lane.

[0005] The embodiments of the present disclosure relate to navigation road and perception lane matching for autonomous and semi-autonomous systems and applications. Systems and methods are disclosed that use geometric similarity to identify or recognize examples of lanes that match roads or portions thereof for use in localization, navigation, and / or other uses by autonomous vehicles, semi-autonomous vehicles, robots, and / or other types of objects or machines.

[0006] In contrast to conventional systems such as those described above, in some embodiments, lane-road matching is performed using geometric similarity to generate effective lane-road assignments that can be used, among other things, for lane planning and decision-making in autonomous or semi-autonomous systems and applications. For example, using techniques provided herein, lane-road assignments can be generated that can be used to generate effective lane-level instructions and for use by a behavior planning module to perform efficient and effective navigation. Furthermore, lanes can be annotated with road-level attributes from a road graph, including road curvature values and traffic-regulating signs and signals, resulting in more detailed and effective information.

[0007] The embodiments described herein are generally directed to matching a lane with a road or portions thereof and generating a lane-road assignment representing the match. In some embodiments, a lane and a road or portions thereof may be identified as matching based on a geometric similarity analysis. In particular, a lane may be identified as matching a road or portions thereof if certain matching conditions specifying geometric similarity are met. For example, matching conditions indicating a match of a lane with a road or portions thereof may include the lane being within a distance threshold and / or a direction threshold of the road.Upon determining that a geometric similarity between a lane and a road, or portions thereof, indicates a match, a match score may be generated to indicate a degree of such geometric similarity. Since a lane may be determined to match multiple roads (e.g., as part of meeting match conditions), the match score may be used to select a particular road to assign to the lane. Thus, a particular lane that matches a road, or portions thereof, may be selected in association with a highest match score for a lane-to-road assignment. Therefore, a lane-to-road assignment generally assigns one or more particular lane segments to one or more road segments that maximize geometric similarity (e.g., based on the match score(s)).The lane-to-road assignment can be represented in several ways. In some cases, the lane-to-road assignment can be represented using one or more lane segment identifiers and one or more road segment identifiers corresponding to one or more locations where the lane is determined to coincide with a road.

[0008] In operation, to identify whether a lane coincides with a road or portions thereof, various points along a lane and a road may be analyzed to determine that a set of matching conditions is met, indicating the geometric similarity between the lane and the road. An example of a matching condition includes that a lane is within a distance threshold of a road. For example, if a lane is within a predetermined distance of a road, the lane may be considered to coincide with the road. In some cases, a perpendicular span extending from a lane and / or a road may be analyzed (e.g., across a lane graph and / or road graph) to identify whether a distance condition is met.For example, a perpendicular line can be extended from a road being analyzed until it intersects a lane being analyzed. A distance from the road to the intersection point can be determined and compared to the distance threshold. In cases where the distance from the road to the intersection point is equal to or less than the distance threshold, the distance condition can be assumed to be met at that particular location. As another example, a perpendicular span can be extended from the road with the length of the distance threshold. In cases where there is an intersection between the perpendicular span and the lane, the distance condition can be assumed to be met at that particular location.

[0009] Another example of a matching condition includes a lane being within a direction threshold of a road. For example, if a lane is in a predetermined direction relative to a road, the lane may be considered to be consistent with the road. In some cases, a similarity between the direction of a lane at a particular location and the direction of a road at a particular location may be determined using a dot product function, as described herein, to identify whether a direction condition is satisfied.

[0010] Therefore, the techniques described herein can be used to match a lane with a road or portions thereof and generate a lane-road assignment representing the lane-road match. The generated lane-road assignment can be provided to a driving stack for autonomous or semi-autonomous vehicles to assist in performing one or more operations related to localization, safe planning, and / or control of the vehicle. Therefore, lane-road assignments can assist an autonomous or semi-autonomous vehicle or other machine type (e.g., robot, construction vehicle, drone, etc.) in navigating a physical environment, and in particular, can assist in lane inference and planning for more accurate and reliable navigation.

[0011] The disclosure extends to all novel aspects or features described and / or illustrated herein.

[0012] Further features of the disclosure are characterized by the independent and dependent claims.

[0013] Any feature in one aspect of the disclosure may be applied to other aspects of the disclosure in any suitable combination. In particular, method aspects may be applied to device or system aspects, and vice versa.

[0014] Furthermore, features implemented in hardware may also be implemented in software, and vice versa. Any reference to software and hardware features in this description should be construed accordingly.

[0015] Any system or device feature described herein may also be provided as a method feature, and vice versa. Functionally described system and / or device aspects (including means and function features) may alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0016] It should also be understood that certain combinations of the various features described and defined in any aspects of the disclosure may be implemented and / or provided and / or used independently of one another.

[0017] The disclosure also provides computer programs and computer program products comprising software code configured to, when executed on a data processing device, perform any of the methods described herein and / or embody any of the device and system features described herein, including any or all component steps of a method.

[0018] The disclosure also provides a computer or computing system (including networked or distributed systems) having an operating system that supports a computer program for performing any of the methods described herein and / or for embodying any of the apparatus or system features described herein.

[0019] The disclosure also provides a computer-readable medium having stored thereon one or more of the aforementioned computer programs.

[0020] The disclosure also provides a signal carrying one or more of the aforementioned computer programs.

[0021] The disclosure extends to methods and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0022] Aspects and embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present systems and methods for aligning navigation road and perception lane for autonomous and semi-autonomous systems and applications are described in detail below with reference to the attached drawings. They show: Fig. 1 is a data flow diagram illustrating an example process for a lane-road matching system, according to some embodiments of the present disclosure; Fig. 2 is a data flow diagram illustrating an example process for facilitating lane-road matching, according to some embodiments of the present disclosure; Fig. 3 is an illustration of an example of a cover frustum relative to vehicle position, according to some embodiments of the present disclosure; Fig. 4A-4B illustrate an effect of incorporating a prologue intersection modification, according to some embodiments of the present disclosure; Fig. 5 illustrates an example of generating a match score value, according to some embodiments of the present disclosure; Fig. 6 to 8 illustrate exemplary methods for generating a lane-road assignment, according to some embodiments of the present disclosure; Fig. 9A is an illustration of an exemplary autonomous vehicle, according to some embodiments of the present disclosure; Fig. 9B shows an example of camera locations and fields of view for the example autonomous vehicle from Fig. 9A, according to some embodiments of the present disclosure; Fig. 9C is a block diagram of an exemplary system architecture for the exemplary autonomous vehicle of Fig. 9A, according to some embodiments of the present disclosure; Fig. 9D is a system diagram for communication between one or more cloud-based servers and the example autonomous vehicle of Fig. 9A, according to some embodiments of the present disclosure; Fig. 10 is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure; and Fig. 11 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0024] Systems and methods relating to matching navigation roads and perception lanes for autonomous and semi-autonomous systems and applications are disclosed. For example, systems and methods are disclosed that use geometric similarity to identify or recognize examples of lanes that match roads or portions thereof for use in localization, navigation, and / or other uses by autonomous vehicles, semi-autonomous vehicles, robots, and / or other types of objects or machines. In this regard, the present techniques can be used to enhance the use of lane graphs. In particular, the present techniques for matching a lane to a road or portions thereof enable more effective lane planning and decision making.For example, using the techniques provided herein, effective lane-level instructions can be generated and used by a behavior planning module to perform efficient and effective navigation. Furthermore, lanes can be annotated with street-level attributes from a road graph, including road curvature values and traffic control signs and signals, resulting in more detailed and effective information.

[0025] Although the present disclosure is described with respect to an exemplary autonomous vehicle or a semi-autonomous vehicle or semi-autonomous machine 900 (alternatively referred to herein as “vehicle 900,” “ego-vehicle 900,” “machine 900,” or “ego-machine 900,” an example of which is described with reference to Fig. 9A-9D), this is not intended to be a limitation. The systems and methods described herein may be used by, without limitation, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more advanced driver assistance systems (ADAS)), autonomous vehicles or machines, guided and unguided robots or robotic platforms, warehouse vehicles, off-highway vehicles, vehicles coupled to one or more trailers, hydrofoils, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely operated vehicles such as drones, and / or other types of vehicles.Furthermore, although the present disclosure may be described in terms of aligning navigation road and perception lane for autonomous or semi-autonomous systems and applications, this is not to be construed as limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology areas where lane detection, localization, and / or navigation may be used.

[0026] Generally, the embodiments described herein are directed to matching a lane with a road (or other delimited type of region, such as a path, a portion of a corridor or walkway, a portion of an open space, etc., with a navigable surface), or portions thereof, and generating a lane-road assignment representing the match. In some embodiments, a lane and a road, or portions thereof, may be identified as matching based on a geometric similarity analysis. In particular, a lane may be identified as matching a road, or portions thereof, if certain matching conditions specifying geometric similarity are met.For example, match conditions indicating a match of a lane with a road or portions thereof may include that the lane be within a distance threshold from the road and that the lane be within a direction threshold from the road. Upon determining that a geometric similarity between a lane and a road or portions thereof indicates a match, a match score may be generated to indicate a degree of such geometric similarity. Since a lane may be determined to match multiple roads (e.g., as part of meeting match conditions), the match score may be used to select a particular road to assign to the lane.For example, a particular lane that coincides with a road or portions thereof may be selected for a lane-road assignment in association with a highest match score. Therefore, a lane-road assignment generally assigns one or more particular lane segments to one or more road segments that maximize geometric similarity (e.g., based on the match score(s).) The lane-road assignment may be represented in several ways. In some cases, the lane-road assignment may be represented using one or more lane segment identifiers and one or more road segment identifiers corresponding to one or more locations where the lane is determined to coincide with a road.

[0027] In operation, to identify whether a lane coincides with a road or portions thereof, various points along a lane and a road may be analyzed to determine that a set of matching conditions is met, indicating the geometric similarity between the lane and the road. An example of a matching condition includes that a lane is within a distance threshold of a road. For example, if a lane is within a predetermined distance of a road, the lane may be considered to coincide with the road. In some cases, a perpendicular span extending from a lane and / or a road may be analyzed (e.g., across a lane graph and / or road graph) to identify whether a distance condition is met.For example, a perpendicular line can be extended from a road being analyzed until it intersects a lane being analyzed. A distance from the road to the intersection point can be determined and compared to the distance threshold. In cases where the distance from the road to the intersection point is equal to or less than the distance threshold, the distance condition can be assumed to be met at that particular location. As another example, a perpendicular span can be extended from the road with the length of the distance threshold. In cases where there is an intersection between the perpendicular span and the lane, the distance condition can be assumed to be met at that particular location.

[0028] Another example of a matching condition includes a lane being within a direction threshold of a road. For example, if a lane is in a predetermined direction relative to a road, the lane may be considered to be consistent with the road. In some cases, a similarity between the direction of a lane at a particular location and the direction of a road at a particular location may be determined using a dot product function, as described herein, to identify whether a direction condition is met.

[0029] The lane-road match analysis may be performed in association with various points or locations along a lane and / or road being analyzed. As one example, the analyzed points may be located a specific distance from each other (e.g., in increments of five meters apart). In some embodiments, the lane-road match may be performed in an iterative manner. In this regard, the analysis may proceed along the length of a lane and / or road to identify whether a set of match conditions is met at the specific location. Therefore, at each incremental position along the road and / or lane, a set of match conditions may be analyzed in association with the analyzed lane-road pair.

[0030] Since various road segments may be present, when completing a progression associated with a road segment, a determination can be made whether the road has unvisited successor road segments. In particular, all unvisited successor road segments along a current path of the road graph traversal can be identified. In cases where a successor road segment is present, the progression can continue along the successor road segment. This iterative progression allows for any number of road segments to be brought into alignment.

[0031] In some embodiments, the iterative process of performing a match analysis for a lane and a road, or portions thereof, may continue until a termination event is identified. In this regard, the match analysis may proceed along sequential points of a road and / or lane until a termination event is identified. One example of a termination event is the identification of an end of a lane and / or road being analyzed. Another example of a termination event is the identification of a failed match. For example, suppose a match analysis performed in association with a particular point along a road results in the failure to satisfy a match condition, such as a distance condition and / or direction condition.In such a case, the failure to satisfy a matching condition can be considered a termination event, allowing the matching analysis to be terminated with respect to the currently analyzed lane-road pair. In some cases, matching is only terminated when a series of matching condition failures is identified (e.g., associated with consecutive points along the road).

[0032] In addition to identifying a match associated with a lane and a road, or portions thereof, a match score may be determined that indicates a degree of match. In some embodiments, a match score may be based on geometric similarity. Because range calculations may be computationally intensive and real-time performance is desired in an ego machine, an approximation of range-based geometric similarity may be used, based on distances between the lane and the road at regularly sampled locations along the length of the road. For example, given a lane and a road, a geometric similarity between a given portion of a lane and a given portion of a road may equal the sum of the inverses of the distances between a lane and a road at sampled locations.Therefore, the better matched or the closer the section of the lane is with respect to the section of the road, the higher the geometric similarity and therefore the higher the quality of the corresponding match.

[0033] Based on the match score values, a lane-road map may be generated and / or provided for a particular lane-road match with the highest match score value. A lane-road assignment generally assigns one or more lanes to one or more particular roads or portions thereof, attempting to maximize or maximizing geometric similarity (e.g., based on the match score value). In some embodiments, a lane-road map is represented via one or more entries indicating the matched portions. In some cases, a lane-road assignment may be represented using one or more lane portion identifiers and one or more road portion identifiers corresponding to locations where the lane is determined to match a road.

[0034] Therefore, the techniques described herein can be used to match a lane with a road or portions thereof and generate a lane-road assignment representing the lane-road match. The generated lane-road assignment can be provided to a driving stack for autonomous or semi-autonomous vehicles to assist in performing one or more operations related to localization, safe planning, and / or control of the vehicle. Therefore, lane-road assignments can assist an autonomous or semi-autonomous vehicle or machine in navigating a physical environment, and in particular, can assist in lane inference and planning for more accurate and reliable navigation.In contrast to conventional approaches, various embodiments provide a way to enable effective lane-level instructions for a behavior planning module to act upon, enabling more precise lane selections and navigation instructions than conventional methods. For example, some embodiments enable the identification of a lane that follows a road furthest along the navigation road graph. Furthermore, various embodiments can be used to annotate lanes with street-level attributes from the navigation road graph, thereby enhancing lane graphs and their use.

[0035] Referring to Fig. 1 is Fig. 1 is a dataflow diagram illustrating an example process 100 for a lane-road matching system, according to some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are set forth as examples only. Other arrangements and elements (e.g., engines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions performed by entities described herein may be performed by hardware, firmware, and / or software.For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may be implemented using similar components, features, and / or functionality as those of the exemplary autonomous vehicle 900 of FIG. Fig. 9A to 9D, the exemplary computing device 1000 of Fig. 10 and / or the exemplary data center 1100 from Fig. 11.

[0036] Fundamentally, the process 100 utilizes a lane-road matching manager 118, which may be configured to match or align a lane or a portion thereof (may also be referred to as one or more lane segments) with a road or a portion thereof (may also be referred to as one or more road segments) and generate a lane-road map 120. In the embodiments described herein, a lane and a road, or portions thereof, may be identified as matching based on a geometric similarity analysis. In particular, a lane may be identified as matching a road or portions thereof when certain matching conditions specifying geometric similarity are met.For example, match conditions indicating a match of a lane with a road or portions thereof may include that the lane be within a threshold distance from the road and that the lane be within a threshold direction from the road. Upon determining that a geometric similarity between a lane and a road or portions thereof indicates a match, a match score may be generated to indicate a degree of such geometric similarity. Since a lane may be determined to match multiple roads (e.g., as part of meeting match conditions), the match score may be used to select a particular road to assign to the lane.For example, a particular lane that coincides with a road or portions thereof may be selected for a lane-road assignment in association with a highest match score. Therefore, a lane-road assignment generally assigns one or more particular lane segments to one or more road segments that maximize geometric similarity (e.g., based on the match score(s).) The lane-road assignment may be represented in several ways. In some cases, the lane-road assignment may be represented using one or more lane segment identifiers and one or more road segment identifiers corresponding to one or more locations where the lane is determined to coincide with a road.

[0037] In some embodiments, lane-road matching may be performed by lane-road matching manager 118 using road graph data 108 and / or lane graph data 116. In this regard, a road graph generator 106 may generate a road graph, and the data associated therewith may be input to lane-road matching manager 118 to perform lane-road matching. A road graph or navigation road graph may be a directed graph representing a road network. Road graph data (or road data), such as road graph data 108, may refer to any data used to represent a road graph (e.g., a road graph generated via road graph generator 106). In some cases, a road graph generator 106 is located on the vehicle.Thus, the road graph generator 106 can generate a road graph in real time based on road network data 104 streamed, for example, from one or more road network data sources 102.

[0038] To generate a road graph and / or road graph data, road graph generator 106 may obtain and use road network data 104. Road network data 104 generally refers to any data representing a road network or a portion thereof. Road network data may include, for example, information about the road network infrastructure, such as the topology and connectivity of the road network, the location of traffic lights, speed limits of different road segments, etc.

[0039] Road network data 104 may be obtained in any manner. In some embodiments, road network data 104 is obtained from one or more road network data sources 102. In some cases, the road network data source(s) 102 may be a navigation head unit (e.g., a third-party navigation head unit) that provides or streams road network data encoded in a particular format, such as according to the ADASIS v2 specification. The ADASIS v2 specification refers to a standard for encoding road network data in a format usable by the road graph generator 106.

[0040] As part of obtaining road network data 104, road graph generator 106 may use this data to generate a road graph. As described, a road graph may be a directed graph representing a road network. In a road graph, an edge in the graph may represent a section of road linking a pair of intersections. In this regard, there may be no entry or exit points on the road section other than at the linked intersections. The direction of an edge may correspond to the direction of travel along the road section represented by the edge. An undivided road section or a road section whose direction of travel is reversible may be represented by a pair of edges.A predecessor of a particular edge may represent a road segment that continues on the road segment represented by the particular edge, and a successor of the particular edge represents a road segment on which the road segment represented by the particular edge may continue.

[0041] One or more edges of a road graph may have associated geometry in the form of a sequence of line segments tracing the center of the road segment represented by the edge. In some cases, the sequence may be a single line segment with endpoints corresponding to the endpoints of the segment. In other cases, the sequence may include multiple line segments, tracing, for example, changes in curvature along the length of the segment. The points of the line segments may be represented in a variety of ways. In one embodiment, line segment points may be represented using the World Geodetic System coordinate space (e.g., WGS84).

[0042] A road graph can be generated for any region or area. For example, in some cases, a road graph is generated for a region containing an immediate or near neighborhood of the vehicle and along the immediate or near portion of what is considered a likely path or most likely path. An example of a likely path or most likely path includes a current portion of a route entered into a vehicle head unit. Another example of a likely path or most likely path includes a path derived based on one or more historical routes and / or vehicle telemetry data interpreted to determine driver intent.

[0043] As part of generating a road graph, the road graph generator 106 may provide road graph data 108 to the lane-road matching manager 118. As described herein, the lane-road matching component 118 may use such road graph data to perform the lane-road matching as described herein. Alternatively or additionally, the road graph data 108 may be stored in a data store for later use (e.g., by the lane-road matching manager 118).

[0044] Additionally, as described, lane-road matching may be performed by a lane-road matching manager 118 using lane graph data 116. In this regard, a lane graph generator 114 may generate a lane graph, and the data associated with it may be input to the lane-road matching manager 118 to perform lane-road matching. A lane graph, or perceptual lane graph, may be a directed graph representing a lane network. Lane graph data (or lane data), such as lane graph data 116, may refer to any data used to represent a lane graph (e.g., a lane graph generated via the lane graph generator 114).

[0045] In some cases, a lane graph generator 114 is located in an ego machine (e.g., an autonomous vehicle). In this regard, the lane graph generator 114 may be or include a perception component or module that generates a lane graph in real time based on sensor data 112, for example, streamed from one or more sensors 110 or another component. In general, a perception component may collect information and extract data from the environment (e.g., via one or more sensors).

[0046] To generate a lane graph and / or lane graph data, the lane graph generator 114 may obtain and use sensor data 112. In some cases, sensor data 112 may be preprocessed so that the data is in a format that can be accepted and processed. Sensor data 112 may be obtained from any number and type of sensor(s) 110, such as, without limitation, LiDAR sensors, RADAR sensors, cameras, ultrasonic sensors, and / or other sensor types as described below with respect to the autonomous vehicle 900. For example, the one or more sensors 110 may include one or more sensors 110 of an ego machine—such as,RADAR sensor(s) 960 of autonomous vehicle 900 and the one or more sensors 110 may be used to generate sensor data 112 representing perceptions in the 3D environment associated with one or more ego machines, as well as objects in the 3D environment surrounding the ego machine(s). According to the embodiments described herein, the sensor data 112 is collected in association with any number of ego machines.

[0047] As part of obtaining the sensor data 112, the lane graph generator 114 can use this data to generate a lane graph. As described, a lane graph can be a directed graph representing a lane network. In a lane graph, an edge in the graph can represent a sequence of one or more segments of lanes of one or more linked road segments. The direction of an edge can correspond to the direction of travel along the lane segments represented by the edge, but this is not necessarily the case. A lane segment represented by a given edge can overlap with the lane segment represented by another edge.The lane of the last lane segment of a predecessor of an edge represents a lane that may or may not continue on the lane of the first lane segment represented by the edge, while the lane of the last lane segment of a successor of an edge represents a lane that may or may not continue on the lane of the first lane segment represented by that edge.

[0048] An edge may have associated geometry in the form of a sequence of line segments tracing the center of the lane segment(s) represented by the edge. In some cases, the sequence may be a single line segment with endpoints corresponding to the endpoints of the segment sequence. In other cases, the sequence may include multiple line segments, tracing, for example, changes in curvature along the length of the segments. The points of the line segments may be represented in a variety of ways. In one embodiment, line segment points are represented over a three-dimensional coordinate space corresponding to the vehicle position and its orientation.In particular, the position of the vehicle may form the origin; the x-axis may lie along the vehicle's longitudinal axis, with positive values indicating locations in front of the vehicle; the y-axis may lie along the vehicle's lateral axis, with positive values indicating locations to the left of the vehicle; and the z-axis may lie along the vehicle's vertical axis, with positive values indicating locations above the vehicle. In some examples, the magnitude of the coordinate values may correspond to meters (or another unit of measurement). For example, a coordinate value of (10.5, -5.3, 1.1) is interpreted as 10.5 meters in front of the vehicle, 5.3 meters to the right of the vehicle, and 1.1 meters above the vehicle.

[0049] A lane graph can be generated for any region or area. As an example, a lane graph may correspond to a region containing a frustum with a near plane behind the vehicle and a far plane in front of the vehicle, with the vehicle location toward the rear of the frustum and centered along the long axis of the frustum. With a brief reference to Fig. 3 represents Fig. 3 provides an illustration of an example of a coverage frustum relative to the vehicle position. In this example, region 302 includes a near plane 304 behind the vehicle 306 and a far plane 308 in front of the vehicle. The vehicle position is toward the rear of the frustum and is centered along the longitudinal axis 310 of the frustum.

[0050] As part of generating a lane graph, lane graph generator 114 may provide lane graph data 116 to lane-road matching manager 118. As described herein, lane-road matching manager 118 may use such lane graph data to perform lane-road matching as described herein. Alternatively or additionally, lane graph data 116 may be stored in a data store for later use (e.g., by lane-road matching manager 118).

[0051] After obtaining road graph data 108 and lane graph data 116, the lane-road matching component 118 may perform lane-road matching. In this regard, the lane-road matching manager 118 may align or align a lane or a portion thereof (also referred to as one or more lane segments) with a road or a portion thereof (also referred to as one or more road segments). In some embodiments, a lane-road match is a sequence of consecutive and contiguous segments of a given lane that aligns or aligns with a sequence of roads or road segments that form a simple path in a road graph.A lane and a road, or portions thereof, may be identified as matching based on a geometric similarity analysis. In particular, a lane may be identified as matching a road, or portions thereof, if certain conditions are met, such as the lane being within a threshold distance from the road and within a threshold direction from the road. Upon determining that a lane matches a road, a match score may be generated to indicate a degree of geometric similarity. Since a lane may be determined to match multiple roads (e.g., as part of meeting match conditions), the match score may be used to select a particular lane to be assigned to the road.Thus, a particular lane that matches a road may be selected in association with a highest match number value to create a lane-road assignment 120.

[0052] A lane-road assignment, such as lane-road assignment 120, generally refers to an assignment of one or more lanes to one or more specific roads or portions thereof that maximizes geometric similarity (e.g., based on the match number value). Lane-road assignment 120 may be represented in a variety of ways. In some cases, lane-road assignment 120 may be represented using one or more lane segment identifiers and one or more road segment identifiers corresponding to locations where the lane is determined to match a road. As one example, a lane-road assignment may include a sequence of entries corresponding to matching lane-road segments.An entry may include a lane segment field that specifies the start and end location of a segment along a lane, a road identification field that specifies the road, and a road segment field that specifies the start and end location of a segment along the road.

[0053] The output of the lane-road assignment 120 by the lane-road match manager 118 may be used by the vehicle 900 of Fig. 9A to 9D when performing one or more operations, such as localization, navigation, and / or others. For example, a representation of the lane-road assignment 120 may be used by one or more control components of the vehicle 900, such as an autonomous or semi-autonomous driving software stack 122 running on one or more components of the vehicle 900. Fig. 9A to 9D (e.g., the one or more SoCs 904, the one or more CPUs 918, the one or more GPUs 920, etc.). For example, the vehicle 900 may use this information (e.g., in the case of obstacles) to locate its position on a map, navigate, plan, or otherwise perform one or more operations (e.g., avoiding obstacles or protrusions, staying in lane, changing lanes, merging, splitting, adjusting a suspension system of the ego machine in accordance with the current road surface, applying early acceleration or deceleration based on an approaching surface slope, apportioning, etc.) within the environment.

[0054] In some embodiments, lane-road allocation 120 may be used by one or more layers of autonomous driving software stack 122 (alternatively referred to herein as “driving stack 122”). The driving stack 122 may include a sensor manager (not shown), one or more perception components (e.g., corresponding to a perception layer of the driving stack 122), a world model manager 126, one or more planning components 128 (e.g., corresponding to a planning layer of the driving stack 122), one or more control components 130 (e.g., corresponding to a control layer of the driving stack 122), one or more obstacle avoidance components 132 (e.g., corresponding to an obstacle or collision avoidance layer of the driving stack 122), one or more actuation components 134 (e.g., corresponding to an actuation layer of the driving stack 122), and / or other components corresponding to additional and / or alternative layers of the driving stack 122.The process 100 may, in some examples, be performed at least in part by or in association with the one or more perception components that may forward the layers of the driving stack 122 to the world model manager, as described in more detail herein.

[0055] The sensor manager may manage and / or abstract sensor data from the sensors of the vehicle 900. For example, and with reference to Fig. 9C, the sensor data may be generated (e.g., continuously, at intervals, based on certain conditions) by the one or more LIDAR sensors 964, the one or more RADAR sensors 960, the one or more ultrasonic sensors 962, the one or more stereo cameras 968, other camera(s), and / or other sensors. The sensor manager may receive the sensor data from the sensors in different formats (e.g., sensors of the same type may output sensor data in different formats) and may be configured to convert the different formats into a unified format (e.g., for each sensor of the same type). As a result, other components, other features, and / or other functionality of the autonomous vehicle 900 may use the unified format, thereby simplifying processing of the sensor data.In some examples, the sensor manager may use a unified format to reapply control to the sensors of the vehicle 900, for example, to adjust frame rates or perform gain control. The sensor manager may also update sensor packets or communications corresponding to the sensor data with timestamps to help inform the processing of the sensor data by various components, features, and functionality of an autonomous vehicle control system.

[0056] A world model manager 126 may be used to generate, update, and / or define a world model. The world model manager 126 may use information generated by and received from the one or more perception components of the driving stack 122 (e.g., the locations of detected obstacles). The one or more perception components may include an obstacle perceiver, a path perceiver, a wait perceiver, a map perceiver, and / or one or more other perception components. For example, the world model may be defined, at least in part, based on the possibilities for obstacles, paths, and wait conditions that may be perceived in real-time or near-real-time by the obstacle perceiver, path perceiver, wait perceiver, and / or map perceiver. The world model manager 126 may update the world model based on recently generated and / or received inputs (e.g.,continuously update data) from the obstacle perceiver, path perceiver, wait perceiver, map perceiver and / or other components of the autonomous vehicle's control system.

[0057] The world model can be used to help inform the one or more planning components 128, the one or more control components 130, the one or more obstacle avoidance components 132, and / or the one or more actuation components 134 of the driving stack 122. The obstacle perceiver can perform obstacle perception, which can be based on where the vehicle 900 is permitted or able to travel (e.g., based on the location of drivable or other navigable paths defined by avoiding detected environmental obstacles and / or detected protrusions in the road surface) and how fast the vehicle 900 can travel without colliding with an obstacle (e.g., an object such as a structure, entity, vehicle, etc.) detected by the sensors of the vehicle 900.

[0058] The path perceiver may perform path perception, for example, by perceiving nominal paths available in a particular situation. In some examples, the path perceiver may further consider lane changes for path perception. A lane graph may represent the path or paths available to vehicle 900 and may be as simple as a single path on a highway entrance ramp. In some examples, the lane graph may include paths to a desired lane and / or indicate available changes on the highway (or other road type), or include nearby lanes, lane changes, interchanges, turns, cloverleaf intersections, merges, and / or other information. In some embodiments, the path perceiver may consider one or more lane graphs and / or one or more lane-to-road assignments.For example, the path perceiver can evaluate a reconstructed 3D road surface to identify lane changes and lane merges.

[0059] The wait perceiver may be responsible for determining restrictions for the vehicle 900 as a result of rules, conventions, and / or practical considerations. For example, the rules, conventions, and / or practical considerations may be related to a 3D road surface, traffic signals, multi-way stops, right-of-way controls, merges, toll plazas, gates, police or other emergency personnel, road workers, stopped buses or other vehicles, one-way bridge assignments, ferry docks, etc. Thus, the wait perceiver may be used to identify potential obstacles and implement one or more controls (e.g., slowing down, stopping, etc.) that may not have been possible had one relied solely on the obstacle perceiver. In some embodiments, the wait perceiver may consider a lane graph and / or a lane-to-road assignment.For example, the waiting perceiver may evaluate a reconstructed 3D road surface to identify an approaching lane merge and determine that early acceleration or deceleration is applied and / or to apply it to incorporate the approaching lane merge.

[0060] The map perceiver may include a mechanism to recognize behavior, and in some examples, to determine specific examples of which conventions apply at a particular location. For example, the map perceiver may determine, based on data representing previous trips or travel, that no U-turns are allowed at a certain intersection between certain times, that an electronic sign showing the direction of lane changes varies depending on the time of day, that two traffic lights in close proximity (e.g., barely offset from each other) are associated with different streets, that in Rhode Island, the first car waiting to turn left at a traffic light is against the law if it turns in front of oncoming traffic when the light turns green, and / or other information.The map perceiver can inform the vehicle 900 about static or stationary infrastructure objects and obstacles. The map perceiver can also generate information for the wait perceiver and / or the path perceiver, for example, to determine which traffic light at an intersection must be green for the vehicle 900 to take a particular path.

[0061] In some examples, information may be sent, transmitted, and / or provided by the card perceiver to one or more servers (e.g., to a card manager of one or more servers 978 of Fig. 9D), and information from the one or more servers may be sent, transmitted, and / or provided to the map perceiver and / or a location manager of the vehicle 900. The map manager may include a cloud allocation application located remotely from the vehicle 900 and accessible to the vehicle 900 over one or more networks. For example, the map perceiver and / or the location manager of the vehicle 900 may communicate with the map manager and / or one or more other components or features of the one or more servers to inform the map perceiver and / or the location manager about past and present trips or journeys of the vehicle 900, as well as past and present trips or journeys of other vehicles. The map manager may provide allocation outputs (e.g.,map data) that can be localized by the localization manager based on a particular location of the vehicle 900, and the localized allocation outputs can be used by the world model manager 126 to generate and / or update the world model.

[0062] The planning component(s) 128 may include, among other components, other features, and / or other functionality, a route planner, a lane planner, a behavior planner, and a behavior selector. The route planner may use information from the map viewer, the map manager, and / or the localization manager, among other information, to generate a planned path that may consist of GNSS waypoints (e.g., GPS waypoints), 3D world coordinates (e.g., Cartesian, polar, etc.) indicating coordinates relative to an origin point on the vehicle 900, etc. The waypoints may represent a specific distance in the future for the vehicle 900, such as a number of city blocks, a number of kilometers, a number of feet, a number of inches, a number of miles, etc., which may be used as a target for the lane planner.

[0063] The lane planner may use a lane graph and / or a lane-road assignment, poses of objects within the lane graph or lane-road assignment, and / or a destination point and direction at a future distance from the route planner as inputs. The destination point and direction may be assigned to the best matching drivable point and direction in the lane graph (e.g., based on GNSS and / or compass heading). A graph search algorithm may then be executed on the lane graph and / or lane-road assignment from a current edge to find the shortest path to the destination point.

[0064] The behavior planner may determine the feasibility of basic behaviors of the vehicle 900, such as remaining in the lane or changing lanes left or right, so that the feasible behaviors can be aligned with the lane planner's output of the most desired behaviors. For example, if it is determined that the desired behavior is not safe and / or available, a default behavior may be selected instead (e.g., the default behavior may be remaining in the lane if the desired behavior or changing lanes is not safe). The behavior planner may use a lane-road assignment to facilitate such planning.

[0065] The control component(s) 130 may follow a trajectory or path (lateral and longitudinal) received from the behavior selector of the planning component(s) 128 as closely as possible and within the capabilities of the vehicle 900. The control component(s) 130 may use tight feedback to handle unplanned events or behaviors that are not modeled and / or anything that results in deviations from the ideal (e.g., an unexpected deceleration). In some examples, the control component(s) 130 may use a forward prediction model that uses the control as an input variable and produces predictions that can be compared to the desired state (e.g., the desired lateral and longitudinal path requested by the planning component(s) 128). The control(s) that minimize the deviation may be determined.

[0066] Although the scheduling component(s) 128 and the control component(s) 130 are illustrated separately, this is not intended to be limiting. For example, in some embodiments, the demarcation between the one or more scheduling components 128 and the one or more control components 130 may not be precisely defined. Therefore, at least some of the components, features, and / or functionality attributed to the one or more scheduling components 128 may be associated with the one or more control components 130, and vice versa. This may also apply to each of the separately illustrated components of the driving stack 122.

[0067] The obstacle avoidance component(s) 132 may assist the autonomous vehicle 900 in avoiding collisions with objects (e.g., moving and stationary objects). The obstacle avoidance component(s) 132 may include a computational mechanism at a "primal level" of obstacle avoidance and act as a "survival brain" or "reptilian brain" for the vehicle 900. In some examples, the obstacle avoidance component(s) 132 may be used independently of components, features, and / or functionality of the vehicle 900 required to comply with traffic laws and drive considerately. In such examples, the obstacle avoidance component(s) may ignore traffic laws, rules of the road, and standards of considerate driving to ensure that collisions do not occur between the vehicle 900 and any objects.Therefore, the obstacle avoidance layer may be a separate layer from the road traffic rules layer, and the obstacle avoidance layer may ensure that the vehicle 900 performs only safe actions from an obstacle avoidance perspective. The road traffic rules layer, on the other hand, may ensure that the vehicle adheres to traffic rules and conventions and observes the legal and conventional right-of-way (as described herein).

[0068] In some examples, the drivable or other navigable paths and / or one or more lane graphs or lane graph assignments may be used by the obstacle avoidance component(s) 132 to determine controls or actions to take. For example, the drivable paths may provide the obstacle avoidance component(s) 132 with indications of where the vehicle 900 can maneuver without colliding with objects, protrusions, structures, and / or the like, or at least where no static structures may be present.

[0069] In non-limiting embodiments, the obstacle avoidance component(s) 132 may be implemented as a separate, discrete feature of the vehicle 900. For example, the obstacle avoidance component(s) 132 may operate separately (e.g., in parallel with, before, and / or after) the planning layer, the control layer, the actuation layer, and / or other layers of the driving stack 122.

[0070] Therefore, the vehicle 900 may use this information (e.g., as edges or rails of the paths) to navigate, plan, or otherwise perform one or more operations (e.g., lane keeping, lane changing, merging, splitting, etc.) in the environment. Although the driving stack 122 is illustrated as receiving the lane-road assignment, as can be appreciated, various components associated with the driving stack 122 may be used to facilitate the generation of the lane-road assignment.

[0071] Referring to Fig. 2 is Fig. 2 is a data flow diagram illustrating an exemplary process 200 for facilitating lane-road matching, according to some embodiments of the present disclosure. In some embodiments, the process 200 represents one possible way in which the lane-road matching manager 118 of Fig. 1 matches one or more lanes with one or more roads or parts thereof and / or generates a lane-road allocation therefrom.

[0072] Fig. 2 illustrates a road graph generator 206 that processes road network data 202, and a lane graph generator 214 that processes sensor data 212 into a format that the lane-road match manager 218 can use (input data 205), and feeds the input data 205 to the lane-road match manager 218, which can generate one or more lane-road assignments 220. In some embodiments, the Fig. 2 may be implemented in an ego machine. For example, the road graph generator 206, the lane graph generator 214, and the lane-road manager 218 may operate in an ego machine to provide real-time lane-road assignments that may be used to perform localization (e.g., positioning an ego machine on a map), path planning (e.g., determining candidate routes for an ego machine), and / or decision making. In other embodiments, one or more of the components illustrated in Fig. The components illustrated in Figure 2 can be implemented remotely from an ego machine. For example, the lane-road matching manager can operate in a server that communicates with the ego machine.

[0073] The road graph generator 206 is generally configured to receive road network data 202. The road network data 202 may include various data types associated with a road network. For example, and without limitation, road network data may include information about the road network infrastructure, such as the topology and connectivity of the road network, the location of traffic signals, speed limits of different road segments, etc.

[0074] As described, the road graph generator 206 generates a road graph, and the data associated with it is provided as input data 205 to the lane-road correspondence manager 218. The road graph generator 206 may generate a road graph and / or road graph data in streaming or real-time. For example, when road network data is streamed from a road network data source, the road graph generator 206 may generate a road graph associated with the received road network data. In some cases, the road graph data provided to the lane-road correspondence manager 218 may be or include the generated road graph or a representation thereof.

[0075] The lane graph generator 214 is generally configured to receive sensor data 212. The sensor data 212 may include various data types that provide perceptions associated with an ego machine or the environment in which the ego machine is located.

[0076] As described, the lane graph generator 214 generates a lane graph, and the data associated with it is provided as input data 205 to the lane-road correspondence manager 218. The lane graph generator 214 may generate a lane graph and / or lane graph data in streaming or real-time. For example, when sensor data is streamed from one or more sensors (e.g., associated with the ego machine or a non-ego machine), the lane graph generator 214 may generate a lane graph associated with the received sensor data. In some cases, the lane graph data provided to the lane-road correspondence manager 218 may be or include the generated lane graph or a representation thereof.

[0077] After receiving input data 205 including road graph data and lane graph data by lane-road matching manager 218, lane-road matching manager 218 is configured to perform lane-road matching. As illustrated, lane-road matching manager 218 may include a match initiator 240, a match manager 242, and a map manager 244. As can be appreciated, these components are provided for illustrative purposes, and any number of components may be used to implement the functionality described herein.

[0078] The matching initiator 240 is generally configured to initiate the performance of a lane-road matching operation. In this regard, the matching initiator 240 may identify one or more root lanes and / or root roads to analyze for lane-road matching. In some embodiments, the matching initiator 240 analyzes the input data 205, such as the lane graph data and / or road graph data, to identify one or more root lanes and / or root roads associated therewith. A lane may be identified as a root lane if there is a position along the lane's geometry that is within a predetermined distance from the ego machine's location. As part of identifying a lane as a root lane, a root lane location may also be identified.A root lane location may be a position closest to the location of the ego machine. In some cases, an equal or similar position may be identified. In such a case, the root lane location may be a position closest to the beginning (or other predetermined point) of a lane geometry. Initiating a lane-road matching using one or more root lanes restricts one or more lane-road matches and / or a lane-road assignment to one or more lanes reachable from one or more root lanes.

[0079] A road can be identified as a root road if there is a position along the road's geometry that is within a predetermined distance of the ego machine's location. In some cases, the predetermined distance is the same distance used to identify a root lane. As part of identifying a road as a root road, a root road location may also be identified. In some cases, a root road location may be a position closest to the ego machine's location. In cases where multiple positions are identified as closest to the ego machine's location, a root lane location may be a position closest to the beginning (or other predetermined point) of a lane's geometry.Initiating a lane-road matching using root roads restricts one or more lane-road matches and / or a lane-road assignment to one or more roads reachable from one or more root roads.

[0080] In some embodiments, the match initiator 240 may initialize values associated with a lane-road match. As described herein, a match count value associated with a lane-road match may be generated during the matching process. In this regard, the match initiator 240 may initialize values for match count values that may be adjusted during the matching process. For example, a match count value for a particular lane-road pair may be initialized to 0 or negative infinity. A match indicator may also be initialized to indicate whether a lane-road pair, or portions thereof, match.

[0081] In some cases, the match initiator 240 may also initialize a value for a lane-road assignment. For example, for a lane-road pair, or portions thereof, an assignment may be initialized as empty, indicating that no matches of one or more lane segments with one or more road segments have been identified.

[0082] In some embodiments, the lane graph data and / or the road graph data may be converted or reformatted to perform lane-road matching to align them. As one example, the matching may be performed using geometric calculations corresponding to a coordinate space of the lane graph. As described, in one implementation, the lane graph may be represented or expressed in a three-dimensional coordinate space corresponding to the position and orientation of the ego machine. Therefore, the road graph or data associated with it may be converted into this three-dimensional coordinate space. For example, WGS84 coordinates of the points of the geometries of the roads associated with the road graph may be converted to coordinates corresponding to the position and orientation of the ego machine.In addition to eliminating the need for non-trivial spherical geometric computations, the transformation also advantageously facilitates the performance of lane graph allocation because it is performed with respect to vehicle position and orientation.

[0083] The matching manager 242 is generally configured to perform lane-road matching. As described, lane-road matching identifies a match between a lane and a road, or portions thereof. In this regard, any number of lane segments may be identified that match any number of road segments. Identifying one or more lane segments that match one or more road segments enables enhanced performance of various autonomous vehicle functionalities, such as lane-level instructions (e.g., used by a behavior planning module), tagging lanes with road-level attributes from the road graph (e.g., road curvature values and traffic-regulating signs and signals), among others.

[0084] Lane-road matching can be performed in several ways. In one embodiment, a lane is identified as matching a road if a set of matching conditions is met. A matching condition can generally refer to any condition that can be used to identify a match between a lane and a road, or parts thereof. As described herein, lane-road matching can be performed according to geometric similarity. Thus, if a lane geometrically resembles a road, the lane-road match, or parts thereof, can be identified. Accordingly, a matching condition can correspond to geometric similarity parameters.

[0085] An example of a match condition includes that a lane is within a distance threshold of a road. For example, if a lane is within a predetermined distance of a road, the lane can be assumed to be consistent with the road. A distance threshold can be determined in several different ways. For example, the distance threshold can be based on a default setting, a preset setting, or a user-selected setting. Furthermore, a distance threshold can be represented in several different ways. For example, a certain metric (e.g., meters) can be used to specify a distance in order to use a threshold to determine a match.In some embodiments, a distance threshold is large enough to account for differences in locations, orientations, and granularity between the lane graph and road graph geometries, but not so large as to impair the plausibility of a match.

[0086] In some cases, a perpendicular span extending from a lane and / or a road may be analyzed (e.g., across a lane graph and / or road graph) to identify whether a distance constraint is satisfied. For example, a perpendicular line from a road being analyzed may be extended until it intersects a lane being analyzed. A distance from the road to the intersection point, also referred to as the intersection distance, may be determined and compared to the distance threshold. In cases where the intersection distance from the road to the intersection point is equal to or less than the distance threshold, the distance constraint may be assumed to be satisfied at that particular location. As another example, a perpendicular span may be extended from the road by the length of the distance threshold.A vertical span can be of any length and / or metric. As an example, a vertical span might be ten meters. In cases where there is an intersection between the vertical span and the lane, the distance constraint can be assumed to be met at that particular location. While these examples extend the vertical span from the road to identify intersections with the lane, the vertical span can alternatively be extended from the lane to identify whether there is an intersection with a road and / or to identify a distance to reach a road intersection.

[0087] Another example of a match condition includes a lane being within a direction threshold of a road. For example, if a lane is in a predetermined direction relative to a road, the lane may be considered to be consistent with the road. A direction threshold may be determined in several different ways. For example, the direction threshold may be based on a default setting, a preset setting, or a user-selected setting. Furthermore, a direction threshold may be represented in several different ways. For example, a certain metric (e.g., degrees) may be used to specify a direction threshold used to determine a match.In some embodiments, a distance threshold is large enough to account for differences in locations, orientations, and granularity between the lane graph and road graph geometries, but not so large as to impair the plausibility of a match.

[0088] In some cases, a similarity between the direction of a lane at a particular location and the direction of a road at a particular location can be determined using a dot product function to identify whether a directionality constraint is met. Generally, when two vectors (e.g., a vector associated with a lane and a vector associated with a road) point in the same direction, the dot product is positive. The more two vectors point in the same direction, the larger the dot product resulting between the two vectors. Thus, vectors associated with points or locations on the lane and the road can be identified and used to determine whether a directionality constraint is met. Other methods, such as transcendental functions, can be used to determine directional similarity.

[0089] As an example implementation using a dot product function, the dot product can be used to determine direction given a road R and a position r along R, a perceptual lane L and a position l along L. Given a feasible intersection of the perpendicular span between R and L (e.g., from r to l), a dot product direction determination test can be employed, calculating the normalized vector representing the direction of R at r and the normalized vector representing the direction of L at l. Subsequently, the dot product of the normalized vectors is determined, and the dot product result is compared to a given threshold, e.g., a threshold of ~0.77, which corresponds to approximately 45 degrees in terms of direction similarity.

[0090] As can be seen, a direction calculation for a lane and / or road (e.g., associated with a vector) may, in some cases, include the calculation of a square root and direction data, such as normalized directions associated with lane and / or road geometries, which may be stored (e.g., cached) for later use. For example, since directions associated with a lane and / or road may be used in future matching constraint analysis, caching normalized directions may enable a more efficient lane-road matching process when needed.

[0091] In some implementations, each match condition of a set of match conditions may need to be met or achieved to identify a match associated with a lane and a road, or parts thereof. For example, suppose a set of match conditions includes a distance condition and a direction condition. In such a case, both the distance condition and the direction condition may need to be met to determine a lane-road match.

[0092] The match manager 242 may perform an analysis of the match conditions associated with various points or locations along a lane and / or road being analyzed. In this regard, various points or locations along a lane and / or road being analyzed may be identified and used to perform a match analysis associated with those points or locations. Such points or locations may be selected in various ways. As one example, the analyzed points may be located a certain distance from each other (e.g., in increments of five meters from each other). For example, assume a lane is being analyzed at a first location, and an intersection of a perpendicular segment extending from the lane to a road is identified at a first location.The first locations associated with the lane and the road can then be analyzed in association with one or more matching conditions. A second lane location located five meters in front of the first lane location can then be identified. An intersection point of a perpendicular segment extending from the lane at this second position to a second road location can then be identified. The second locations associated with the lane and the road can then be analyzed in association with the one or more conditions.

[0093] In some cases, to identify a match between a lane and a road, or parts thereof, a threshold number of points or a threshold distance between points can be analyzed and identified as matches. For example, suppose only two sequential points between a lane and a road are identified as matching. In such a case, a lane-road match may not be determined even though two points along the lane have been identified as matching the road (e.g., satisfying match conditions).

[0094] In some embodiments, the matching manager 242 performs the lane-road matching in an iterative manner. In this regard, the matching manager 242 may progress along the length of a lane and / or road to identify whether a set of matching conditions is met at the particular location. Therefore, at each incremental position along the road and / or lane, a set of matching conditions may be analyzed in association with the analyzed lane-road pair.

[0095] As an example, at each point in progression along the road and / or lane, an intersection point between a lane and a vertical span of the road at a location is identified. The vertical span at a road location may be a line segment centered along the road at the location and perpendicular to the direction of the road at the location. In some cases, the vertical span may extend in either direction. For example, the length of the vertical span may be twice the radius of the vertical span. The result of the intersection point (e.g., between the lane and the vertical span extending from the road) is evaluated against a set of match conditions to determine whether a match exists or not.An example condition is that there is a unique intersection location along the lane that is greater than or equal to the previous lane location. Another example match condition is that the direction of the lane at the intersection location is sufficiently similar to the direction of the road at the location according to a directional similarity threshold. In cases where a match is identified, various parameters may be updated. After identifying a match and updating all parameters, progress along the road proceeds at an incremental amount. In some cases, the incremental amount reflects a walking rate. A higher walking rate indicates a shorter distance between consecutive locations with vertical span along the length of the road graph.A higher walking rate can lead to higher precision and plausibility of a match determination.

[0096] As described herein, different road segments may be used to represent different roads. Therefore, after completing a progression or walk associated with a road segment, a determination may be made if the road has unvisited successor road segments. In particular, all unvisited successor road segments along a current path of the road graph traversal may be identified. In cases where a successor road segment is present, progress may continue along the successor road segment. This iterative progression allows for any number of road segments to be brought into alignment.

[0097] In some embodiments, the iterative process of performing a conformance analysis for a lane and a road, or portions thereof, may continue until a termination event is identified. A termination event may be any event indicating that the conformance analysis for the lane and / or road is to be terminated. In this regard, the conformance analysis may proceed along sequential points along a road and / or lane until a termination event is identified. An example of a termination event is the identification of an end of a lane and / or road being analyzed. For example, suppose an end of a lane segment and / or road segment is identified and no subsequent or successor lane segments and / or road segments are identified.In such a case, the matching analysis can be terminated with respect to the currently analyzed lane-road pair.

[0098] Another example of a termination event is the identification of a failed match. For example, suppose a match analysis performed in association with a specific point along a road fails to satisfy a match condition, such as a distance condition and / or direction condition. In such a case, the failure to satisfy a match condition can be considered a termination event, allowing the match analysis to be terminated with respect to the current lane-road pair being analyzed.

[0099] In some cases, a termination event can only be identified after a threshold number or distance of match condition failures, such as consecutive match condition failures, has been identified. For example, suppose a threshold number of match condition failures is five. In such a case, upon identifying a match condition failure associated with five consecutive points along a current path of a road graph, the termination event can be identified. Implementing a threshold number or distance of match condition failures enables a more efficient process. For example, the iterative matching process can be carried out without the constraint over the entire length of the path of a road graph (which, for example,several hundred meters long) without finding a feasible intersection or match. In operation, the lower the limit, the more powerful the algorithm becomes, but using a limit that is too low can lead to the degradation of the plausibility of the computed matching result. Furthermore, such an approach allows for the tolerance of otherwise coarse-grained geometric road representations of standard-resolution road map data when matching them with fine-grained perceptual lane geometries (e.g., particularly at intersections with a rectangular arrangement of road geometries, while perceptual lanes can accommodate gentle curvatures of the intersection lanes).

[0100] In some embodiments, the match manager 242 may be configured to perform modifications to the intersection in association to identify lane-road matches. For example, and with reference to Fig. 4A to 4B represents Fig. 4A to 4B provide an illustration of the effect of including a prologue intersection modification. Fig. 4A provides an exemplary correspondence calculation between a root lane l and a root road r without applying the modification, and Fig. Figure 4B provides an example correspondence calculation between the root lane l and the root road r with the modification applied. In Fig. 4A and Fig. 4B, the ego-machine location is marked with a star, the root lane location with a triangle, and the root road location with a circle. Line segments emanating from road r denote the perpendicular spans of a threshold distance along the length of r, where dashed line segments correspond to non-matching results (e.g., one or more matching conditions are not met) and solid line segments correspond to matching results (e.g., matching conditions are met). Fig. 4A, line l begins after road r along the direction of r. As a result, the initial outcome of the intersection is not feasible before advancing to a series of feasible outcomes after advancing the location along r. Consequently, the computed correspondence between l and r exhibits a misalignment at the beginning of the correspondence. With the addition of the prologue intersection modification, if the initial outcome of the intersection between l and r is not feasible, a fallback intersection computation is performed using an instance of the perpendicular span propagated from the root location of r to the root location of l, as in Fig. 4B. If the transferred perpendicular span intersects r at a unique location and the location of the intersection lies between the root location of r and the nearest location along r, the result of the intersection is considered feasible, resulting in the elimination of the misalignment described above.

[0101] Referring to Fig. 4C to 4D represents Fig. 4C to 4D provide an illustration of the effect of including an epilogue intersection modification. Fig. Figure 4C provides an exemplary correspondence calculation between a terminal lane l and a road r without applying the modification, while Fig. 4D provides an example applying the modification. Line segments emanating from r denote the perpendicular spans of the intersection result calculations along the length of r, where dashed line segments correspond to infeasible match results and solid line segments correspond to feasible match results. If the match calculation in this example ends with a sequence of infeasible results, a fallback intersection calculation can be performed using an instance of the perpendicular span of r that is the road with the most recent feasible match result propagated to the end of l. If the propagated perpendicular span intersects r at a unique location and the intersection location follows the last feasible result location, i.e.along r, then this is considered feasible, resulting in the elimination of the misalignment described above.

[0102] In some implementations, linear scans can be used to perform matching analysis. In other implementations, a spatial index can be used to perform matching analysis. A spatial index can limit the number of geometries evaluated. The spatial index is initialized at the beginning of the matching procedure with the geometries of the lanes reachable from one or more root lanes. In practice, the upfront costs associated with creating the spatial index are expected to be amortized over the course of the algorithm's execution, ultimately enabling a significant reduction in the number of intersection calculations performed by the algorithm.

[0103] In addition to identifying a match associated with a lane and a road, or portions thereof, the match manager 242 may be configured to generate a match score to indicate a degree of match. In some embodiments, a match score may be based on geometric similarity. Because range calculations can be computationally intensive and real-time performance is desired in an ego machine, a range-based geometric similarity approximation based on distances between the lane and the road at regularly sampled locations along the length of the road may be used.For example, given a lane and a road, the geometric similarity between a given segment of a lane and a given segment of a road can be equal to the sum of the inverses of the distances between a lane and a road at sampled locations. In other words, the better matched, or the closer, the segment of the lane is with respect to the segment of the road, the higher the geometric similarity and therefore the higher the quality of the corresponding match.

[0104] To determine a reciprocal of a distance between a lane location and a road, in some cases, a perpendicular span extending from the road and / or the lane may be used. As an example, a perpendicular span may be extended from a road point until the perpendicular span intersects the lane or reaches a distance threshold. Such an intersection distance may be subtracted from the distance threshold to generate a reciprocal of a distance between a lane location and a road. For example, suppose a perpendicular span is extended from a road to intersect a lane, and such an intersection distance is identified as 0.2 units. Also suppose a distance threshold is 1.0 units.In such a case, the inverse of the distance is 1.0 to 0.2, resulting in a match score of 0.8 for this particular location. The relatively high match score of 0.8 indicates a high degree of similarity at this particular location.

[0105] In some cases, a match score may be determined at sampled locations (e.g., at each sampled location). For example, after progressing to a next point along a road and performing a match analysis, a match score associated with it may be determined. In some cases, the match score determined for the sampled location may be specific to that location (e.g., an inverse of the distance between the lane point and the road point being analyzed). In other cases, the match score determined for the sampled location may be an aggregate or sum up to that point (e.g., a sum of the inverses of the distances associated with the current sampled location and previous sampled locations along the path). Additionally or alternatively, a match score may be determined after identifying a terminal event (e.g.,the end of a lane or road being analyzed). For example, as described above, a match score value can be determined when a match is no longer identified (e.g., within a threshold distance or a consecutive number of points) or when the end of a lane or road is identified. In this example, when a terminal point along the matching process is identified, the match score value can be determined.

[0106] The map manager 244 is generally configured to manage the generation of a lane-road map. As described herein, a lane-road map provides an indication of a match or alignment of a lane and a road. In some embodiments, a lane-road map is represented via one or more entries that indicate the aligned portions. A lane-road assignment, such as lane-road assignment 220, may be an assignment of one or more lanes to one or more specific roads or portions thereof that maximize geometric similarity (e.g., based on the match number value). The lane-road assignment 220 may be represented in various ways.In some cases, the lane-to-road assignment 120 may be represented using one or more lane segment identifiers and one or more road segment identifiers corresponding to locations where the lane is determined to coincide with a road. As an example, a lane-to-road assignment may include a sequence of entries corresponding to matching lane-to-road segments. An entry may include a lane identity field indicating a lane, a lane segment field indicating the start and / or end location of a segment along a lane, a road identification field indicating the road, and a road segment field indicating the start and / or end location of a segment along the road. Therefore, different entries may correspond to different lane segment and road segment matches.For example, suppose a lane is identified as matching a first road segment. In such a case, a first entry may indicate a match between the lane and the first road segment. Suppose the iterative matching process continues and identifies that the lane matches a second road segment following the first road segment. In such a case, a second entry may indicate a match between the lane and the second road segment.

[0107] As described, in some cases, multiple lane-road matches may be identified. For example, a lane may be identified as matching one road path and as matching another road path (e.g., based on satisfying match conditions). In such a case, the map manager 244 may identify or select a lane-road match with the best fit to generate a lane-road assignment. In this regard, the match score values may be used to select which lane-road match to use to generate a lane-road assignment. For example, as described, a higher match score value may represent greater geometric similarity.Accordingly, to maximize geometric similarity for a lane-road assignment, a lane-road match with the highest match score may be selected and used to generate a lane-road assignment. In other embodiments, lane-road assignments may be generated in association with different lane-road matches, and the assignment associated with a highest match score may be selected for output or use in performing various autonomous vehicle functionalities, such as lane-level instructions (e.g., used by a behavior planning module), assigning lanes with road-level attributes from the road graph (e.g., road curvature values and traffic control signs and signals), among others.

[0108] In some embodiments, map manager 244 may perform various post-processing operations. For example, consider a lane with a match that includes a portion of road r, followed by a portion of road r', where road r' is a successor to road r. Since entries of the computed match may be based on specific locations or points of lane and road match results, it is possible that the entry referring to r defines a portion of r that does not end at the end of r, and additionally, it is also possible that the entry referring to r' defines a portion of r' that does not begin at the beginning of r'. Likewise, it is possible that these entries define portions of lane l that are not contiguous for the entire length of lane l.For example, a post-processing step can be used to ensure that the assigned sections of lane l are contiguous along the entire length of lane l. As an example, map manager 244 can apply heuristics to address scenarios where there is uncertainty about the locations of the beginning and end of the matching sections.

[0109] Just as an example, Fig. 4 provides an illustration of the lane-road correspondence. In Fig. 4, a lane l is analyzed for correspondence with a section of a road graph containing four roads: r a , r b , r c and r dLine segments extending along the length of the road denote the vertical spans with a certain distance threshold. In cases where the vertical span intersects the lane, the road can be identified as a match with the lane. Fig. 4, the solid segments indicate matches, while the dashed segments correspond to mismatch results. During execution, the algorithm analyzes a lane geometry with respect to a geometry associated with paths (e.g., a sequence of streets) of the road graph. In this example, a first road path r contains a , r b , r d and a second road path contains r a , r cAn iterative approach can be used, initially proceeding along the road paths to perform a geometric similarity analysis. This allows for lane-to-road correspondence analysis at positions along the roads corresponding to the perpendicular line segments.

[0110] With reference to Fig. 5 represents Fig. 5 provides an example of generating a match number value. Suppose the first street r a is analyzed, for example, and the result is a match value of 1. Now assume that street r dis analyzed for a match score value. Initially referring to line segment 502, assume that the radial distance threshold to which line segment 504 extends is 1. In some cases, threshold 1 may represent a distance or a normalized value. Now assume that line segment 502 is determined to intersect the lane at 0.7. In such a case, the match score value may be determined to be 0.3 (1 - 0.7). The new aggregate match score for the path is now 1.3. Next, line segment 506 may be analyzed, again yielding a corresponding match score value of 0.3 (1 - 0.7). In such a case, the new aggregate match score for the path is now 1.6.Suppose such an iterative process continues over line segment 508, yielding an aggregate match score of 1.6, since the remaining line segments appear to intersect at the radial distance threshold, yielding corresponding match scores of 0. In this example, since the match score is 1.6 of path r. a , r b , r d greater than the match number value 1 of r a , r c is, the path r a , r b , r d selected to produce a lane-road map.

[0111] Below, various example algorithms are provided to exemplify approaches to implementing the technology described herein. In these examples, a basic matching algorithm and a general matching algorithm are provided. The basic matching algorithm is generally restricted to lane graphs where lanes have no successors, while the general matching algorithm is an extension of the basic matching algorithm that removes the limitation of the basic matching algorithm. In this respect, the general matching algorithm can be used when lanes have successors. As can be seen, the general matching algorithm is applicable even when lanes have no successors.

[0112] First, a basic matching algorithm is provided that implements the basic matching algorithm:

[0113] In this implementation, the method begins by initializing an allocation and an associated match score for each lane. In this example, the allocation is initialized to empty, and the match score is initialized to negative infinity. For each root lane l and each root road r, the method invokes the basic recursive matching method described below. Each call to MatchRecursive results in the execution of an acyclic depth-first traversal of the road graph, starting at r. During the traversal, an allocation and a corresponding match score applicable to l for the sequence of roads spanning the current path in the road graph are computed. At the beginning of the traversal, the computed allocation is empty, while the computed match score is equal to negative infinity.Each time the traversal reaches a terminal street, if the calculated match score is greater than the match score of l, the allocation of l and the match score of l are updated to the calculated allocation and the calculated match score, respectively.

[0114] An implementation of a basic matching algorithm to perform a recursive or iterative matching approach can be as follows:

[0115] In this algorithm, the method includes an identity parameter denoted by r, a road location parameter denoted by p, an allocation parameter denoted by M, and a match number value denoted by µ. The method begins by initializing an allocation sequence entry e by setting the start and end locations of the lane segment field to λ, the road identity field to r, and the start and end locations of the road segment field to ρ. After initialization, e is shifted to M. Next, as long as p is less than the length of r, the method proceeds along the length of r at a regular rhythm called the walk rate. At each step of walking, a result of the intersection between l and the perpendicular span of r at location p is computed.The perpendicular span of r at location p can be a line segment centered at location p along r and perpendicular to the direction of r at location p. The length of the perpendicular span can be twice the radius of the perpendicular span. The result of the intersection can be evaluated against a pair of constraints to determine whether it is a match or not.

[0116] In this example, a first condition is that there is a unique intersection location along l that is greater than or equal to λ. The second condition is that the direction of l at the intersection location is sufficiently similar to the direction of r at location p according to a direction similarity threshold. If the intersection result is identified as a match, it is processed. A match result can be processed by setting λ to the intersection location along l; setting the lane segment field ending location of e to the intersection location along l; setting the road segment field ending location of e to p; and incrementing µ by the difference between the radius of the perpendicular span and the distance between the intersection location along l and location p along r.After calculating the match score and, if applicable, processing it, walking proceeds along r by incrementing p by an amount reflecting the walking rate. If r has unvisited successors after completing walking, e.g., along the current path of the road graph traversal, the method recursively invokes itself for each successor r' of r that remains to be visited, using l for the lane identity parameter, λ for the lane location parameter, r' for the road identity parameter, ρ - length(r) for the road location parameter, M for the allocation sequence parameter, and µ for the match sequence number value parameter. If r has no unvisited successors, the method processes the calculated match score by examining µ and the match number value of l.If µ is greater than the match score of l, the match score of l is updated to M while the match score of l is updated to µ. At the end of the procedure, e is extracted from M.

[0117] Regarding the general matching algorithm, the following example provides a matching method that implements the general matching algorithm:

[0118] This approach is generally similar to the matching procedure of the basic matching algorithm. This approach invokes a version of the general recursive matching procedure, resulting in the execution of an acyclic depth-first traversal of the road graph starting at r and an acyclic depth-first traversal of the lane graph starting at l. During the traversal, an allocation and corresponding match score can be calculated, applicable to the lanes in the sequence of lanes spanning the current path in the lane graph, for the sequence of roads spanning the current path in the road graph.Each time the traversal reaches a terminal road, for each lane l in the current path in the lane, if the computed match score value is greater than the match score value of l, then the allocation of l is updated to a subsequence of entries in the computed allocation that are specific to l, while the match score value of l is updated to the computed match score value.

[0119] An example of the general matching algorithm for performing a recursive or iterative matching approach is provided below:

[0120] This approach is generally similar to the corresponding approach of the basic matching algorithm. In this approach, entries of the current allocation can be extended by adding a lane identity field. The lane identity field indicates that the entry is specific to a given lane. At the beginning of the procedure, when e is initialized, the lane identity field of e can be set to l. After completing walking r, if r has no unvisited successors, the procedure processes the computed matching result by examining the match score value of each lane l' for which there are one or more entries in the computed match whose lane field is l'.For each such lane l', if µ is greater than the match score of l', the match of l is updated to the subsequence of entries in the computed match whose lane field is equal to l', while the match score of l' is updated to µ.

[0121] Furthermore, with this approach, if the computed intersection result is not identified as a match, the method proceeds to call MatchRecursive recursively for each lane that is an unvisited successor of l. In other words, if there is no matching intersection result on the current lane, the method proceeds to attempt to generate matching intersections with lanes in the unvisited subpaths of the lane graph rooted in the lane.

[0122] With reference Fig. 6-8, each block of the methods 600, 700, and 800 described herein comprises a computational process that may be performed using any combination of hardware, firmware, and / or software. Various functions may be performed, for example, by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media. The methods may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. Furthermore, the methods 600, 700, and 800 may be implemented, for example, with respect to the lane-road matching system of Fig. 1. However, these methods may additionally or alternatively be performed by any system or combination of systems, including, without limitation, the systems described herein.

[0123] Fig. 6 is a flowchart illustrating a method 600 for generating a lane-to-road assignment, according to some embodiments of the present disclosure. The method 600, at block B602, includes receiving road data representing at least one road segment and lane data representing a lane associated with an ego machine location. The road data representing the one or more road segments may be road graph data, and the lane data representing the lane may be lane graph data.

[0124] The method 600, at block B604, includes determining that one or more consecutive road segments coincide with a travel lane based at least on a geometric similarity between the travel lane and the one or more consecutive road segments. Determining whether one or more road segments coincide with a travel lane based on a geographic similarity may be performed in various ways. As one example, such a determination may be based on the travel lane being positioned within a distance threshold of one or more points along the one or more consecutive road segments.In some cases, determining whether the travel lane is positioned within a distance threshold may include extending perpendicular line segments from one or more points along the one or more consecutive road segments to the intersection with the travel lane. Such a determination may be performed in an iterative manner, for example, by progressing along successive points (e.g., positioned a predetermined distance from each other) of the one or more consecutive road segments. Alternatively or additionally, such a determination may be based on the travel lane being positioned within a direction threshold of one or more points along the one or more consecutive road segments. In some embodiments, an extent to which the one or more consecutive road segments coincide with the travel lane is determined.In some cases, such an extent may be generated based on a sum of one or more inverses of distances between the lane and one or more sampled locations along the one or more consecutive road segments.

[0125] The method 600, at block B606, includes generating a representation of the lane assigned to the one or more consecutive road segments based at least on determining that the one or more consecutive road segments have been matched to the lane. In some embodiments, a representation of the lane assigned to the one or more consecutive road segments may be based on a determination to generate the match. For example, a match may be desired if the extent to which the one or more consecutive road segments match the lane is greater than other match extents associated with other road segments.In one embodiment, the representation of the lane assigned to the one or more consecutive road segments includes a representation of a start and end location of a segment along the lane, a representation of an identity of at least a portion of the consecutive road segments, and a representation of a start and end location of at least one road segment.

[0126] Fig. 7 is a flowchart illustrating a method 700 for generating a lane-to-road assignment, according to some embodiments of the present disclosure. The method 700, at block B702, includes determining that a road segment of a road graph associated with an ego machine matches a lane of a lane graph associated with the ego machine based at least on a geometric similarity between the lane and the road segment. Determining whether one or more road segments match a lane based on a geographic similarity may be performed in various ways. As one example, such a determination may be based on the lane being positioned within a distance threshold of one or more points along the one or more consecutive road segments.In some cases, determining whether the lane is positioned within a distance threshold may include extending perpendicular line segments from one or more points along the one or more consecutive road segments to the intersection with the lane. Alternatively or additionally, such a determination may be based on the lane being positioned within a direction threshold of one or more points along the one or more consecutive road segments. In one embodiment, a match determination may be performed in an iterative manner, for example, by progressing along consecutive points (e.g., positioned a predetermined distance from each other) of the one or more consecutive road segments until a termination event is detected.For example, a termination event may be detected when a road or lane ends. As another example, a termination event may be detected when a threshold number (e.g., five) of match failures are consecutively identified along points on the road.

[0127] The method 700, at block B704, includes generating a representation of the extent to which the road segment coincides with the travel lane based at least on the geometric similarity between the travel lane and the road segment. In one embodiment, the representation of the extent to which the road segment coincides with the travel lane is generated based at least on a sum of one or more inverses of one or more distances between the travel lane and one or more sampled locations along the road segment.

[0128] The method 700, at block B706, includes generating a representation of the lane assigned to the road segment based at least on the representation of the extent to which the road segment matches the lane. In some embodiments, a representation of the lane assigned to the one or more consecutive road segments may be based on a determination to generate the assignment. For example, an assignment may be desired if the extent to which the one or more consecutive road segments match the lane is greater than other matching extents associated with other road segments.In one embodiment, the representation of the lane assigned to the one or more consecutive road segments includes a representation of a start and end location of a segment along the lane, a representation of an identity of at least a portion of the consecutive road segments, and a representation of a start and end location of at least one road segment.

[0129] Fig. 8 is a flowchart illustrating a method 800 for generating a lane-to-road assignment, according to some embodiments of the present disclosure. The method 800, at block B802, includes identifying a road segment of a road graph associated with an ego machine and a lane of a lane graph associated with the ego machine to perform a matching analysis. In some cases, the road segment and lane are identified based on proximity to the ego machine.

[0130] The method 800, at block B804, includes determining that the travel lane is within a predetermined distance from a particular point along the road segment. In some cases, a perpendicular span extending from the travel lane and / or the road segment may be analyzed to identify whether a distance condition is met. For example, a perpendicular line may be extended from the road segment being analyzed until it intersects the travel lane being analyzed. A distance from the road to the intersection point may be determined and compared to the predetermined distance. As another example, a perpendicular span may be extended from the road by the distance threshold length. A perpendicular span may have any length and / or metric. As one example, a perpendicular span may be ten meters.In cases where there is an intersection between the vertical span and the lane, it can be assumed that the distance condition is satisfied at that particular location.

[0131] The method 800, at block B806, includes determining that the lane is within a direction threshold of the particular point along the road segment. A direction threshold may be determined in a variety of ways. For example, the direction threshold may be based on a default setting, a preset setting, or a user-selected setting. Furthermore, a direction threshold may be represented in a variety of ways. For example, a particular metric (e.g., degrees) may be used to specify a direction threshold used to determine a match. In some cases, a similarity of a direction of a lane at a particular location and the direction of a road at a particular location may be determined using a dot product function to identify whether a direction condition is met.

[0132] The method 800, at block B808, includes determining that the lane is within a predetermined distance of a particular point along the road segment. For example, if a termination event is identified, a determination may be made to terminate the analysis of subsequent points along the road segment. A termination event may be the identification of an end of a lane and / or road. Another example of a termination event may be a failure to satisfy a match condition or a failure to satisfy a match condition associated with a predetermined number of consecutive points (e.g., five consecutive match failures).In cases where another subsequent point along the road segment can be analyzed, the method returns to blocks B804 and B806 to evaluate the matching conditions for the next point.

[0133] On the other hand, if it is determined that there are no further subsequent points along the road segment to be analyzed, the method 800 proceeds to block B810, where a match score is generated to indicate the extent of match between the lane and the road segment. The match score may be based on geometric similarity. For example, given a lane and a road, a geometric similarity between a given portion of a lane and a given portion of a road may equal the sum of the inverses of the distances between a lane and a road at sampled locations. To determine an inverse of a distance between a lane and a road location, in some cases, a perpendicular span extending from the road and / or the lane may be used.As an example, a vertical span can be extended from a road point until the vertical span intersects the lane or reaches a distance threshold. Such an intersection distance can be subtracted from the distance threshold to generate an inverse of a distance between a lane location and a road.

[0134] Method 800, at block B812, generates a lane-road assignment based on the match score. For example, in some cases, a lane-road assignment may be generated when the match score is greater than other match scores associated with other lane-road segment pairs.

[0135] The systems and methods described herein may be used, without limitation, by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), guided and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, hydrofoils, boats, shuttles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely operated vehicles such as drones, and / or other types of vehicles.Furthermore, the systems and methods described herein may be used for a variety of purposes, including, without limitation, machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actor simulation and / or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, and / or other suitable applications.

[0136] The disclosed embodiments may include a variety of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented with a robot, aviation systems, media systems, boat systems, intelligent area surveillance systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems including one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing light transport simulations,Systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and / or other types of systems. EXEMPLARY AUTONOMOUS VEHICLE

[0137] Fig. 9A illustrates an example autonomous vehicle 900, according to some embodiments of the present disclosure. The autonomous vehicle 900 (alternatively referred to herein as "vehicle 900") may include, without limitation, a passenger vehicle, such as a car, a truck, a bus, an emergency service vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire engine, a police vehicle, an ambulance, a boat, a construction vehicle, an underwater vehicle, a robotic vehicle, a drone, an aircraft, a vehicle coupled to a trailer (e.g., a semi-truck used to transport cargo), and / or another type of vehicle (e.g., that is unmanned and / or accommodates one or more passengers).Autonomous vehicles are generally described in terms of levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) standard "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018, Standard No. J3016-201609, published September 30, 2016, and prior and future versions of this standard). Vehicle 900 may exhibit functionality consistent with one or more of the Level 3 through Level 5 autonomous driving levels.The vehicle 900 may exhibit functionality according to one or more of Level 1 through Level 5 of autonomous driving levels. For example, depending on the embodiment, the vehicle 900 may be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). The term "autonomous" as used herein may include any and / or all types of autonomy for the vehicle 900 or other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, assisted autonomy, semi-autonomous, primarily autonomous, or another designation.

[0138] The vehicle 900 may include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle 900 may include a propulsion system 950, such as an internal combustion engine, a hybrid electric power plant, a pure electric motor, and / or another type of propulsion system. The propulsion system 950 may be connected to a drivetrain of the vehicle 900, which may include a transmission to enable propulsion of the vehicle 900. The propulsion system 950 may be controlled in response to receiving signals from the throttle or accelerator 952.

[0139] A steering system 954, which may include a steering wheel, may be used to steer the vehicle 900 (e.g., along a desired path or route) when the propulsion system 950 is operating (e.g., when the vehicle is moving). The steering system 954 may receive signals from a steering actuator 956. The steering wheel may be optional for full automation (Level 5).

[0140] The brake sensor system 946 may be used to apply the vehicle brakes in response to receiving signals from the brake actuators 948 and / or the brake sensors.

[0141] The one or more controllers 936 that control one or more systems on chips (SoCs) 904 ( Fig. 9C) and / or GPUs may provide signals (e.g., representative of instructions) to one or more components and / or systems of the vehicle 900. For example, the one or more controllers may send signals to apply the vehicle brakes via one or more brake actuators 948, to apply the steering system 954 via one or more steering actuators 956, to apply the propulsion system 950 via one or more throttle / accelerator devices 952. The one or more controllers 936 may include one or more built-in (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 900.The one or more controllers 936 may include a first controller 936 for autonomous driving functions, a second controller 936 for functional safety functions, a third controller 936 for artificial intelligence functions (e.g., computer vision), a fourth controller 936 for infotainment functions, a fifth controller 936 for emergency redundancy, and / or other controllers. In some examples, a single controller 936 may perform two or more of the above functionalities, two or more controllers 936 may perform a single functionality, and / or any combination thereof.

[0142] The one or more controllers 936 may provide the signals to control one or more components and / or systems of the vehicle 900 in response to sensor data received from one or more sensors (e.g., sensor inputs). The sensor data may be received, for example and without limitation, from one or more of the following: global navigation satellite systems (“GNSS”) sensor(s) 958 (e.g., global positioning system sensor(s)), RADAR sensor(s) 960, ultrasonic sensor(s) 962, LIDAR sensor(s) 964, inertial measurement unit (IMU) sensor(s) 966 (e.g., accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s), etc.), microphone(s) 996, stereo camera(s) 968, wide-angle camera(s) 970 (e.g., fisheye cameras), infrared camera(s) 972, ambient camera(s) 974 (e.g.,360-degree cameras), long-range and / or medium-range camera(s) 998, speed sensor(s) 944 (e.g., for measuring the speed of the vehicle 900), vibration sensor(s) 942, steering sensor(s) 940, brake sensor(s) (e.g., as part of the brake sensor system 946), and / or other sensor types.

[0143] One or more of the controllers 936 may receive inputs (e.g., in the form of input data) from an instrument cluster 932 of the vehicle 900 and provide outputs (e.g., in the form of output data, display data, etc.) via a human-machine interface (HMI) display 934, an audible annunciator, a speaker, and / or via other components of the vehicle 900. The outputs may include information such as vehicle speed, RPM, time, map data (e.g., the high-definition (“HD”) map 922 of Fig. 9C), location data (e.g., the location of the vehicle 900, e.g., on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the one or more controllers 936, etc. For example, information about the presence of one or more objects (e.g., a road sign, a warning sign, a changing traffic light, etc.) and / or information about maneuvers that the vehicle has performed, is currently performing, or will perform (e.g., change lanes now, take exit 34B in two miles, etc.) may be displayed on the HMI display 934.

[0144] The vehicle 900 further includes a network interface 922 that may utilize one or more wireless antennas 926 and / or modems for communication over one or more networks. The network interface 922 may, for example, be capable of communication via Long-Term Evolution (LTE), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communications (GSM), IMT-CDMA Multi-Carrier (CDMA2000), etc. The one or more wireless antennas 926 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth Low Energy (LE), Z-Wave, ZigBee, etc.and / or Low Power Wide Area Networks (LPWANs), such as LoRaWAN, SigFox, etc.

[0145] Fig. 9B is an example of camera locations and fields of view for the example autonomous vehicle 900 of Fig. 9A, according to some embodiments of the present disclosure. The cameras and respective fields of view are an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at various locations on the vehicle 900.

[0146] The camera types for the cameras may include, but are not limited to, digital cameras that may be configured for use with the components and / or systems of the vehicle 900. The one or more cameras may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera types may be capable of any image capture rate, such as 60 frames per second (fps), 190 fps, 240 fps, etc., depending on the embodiment. The cameras may use rolling shutters, global shutters, another type of shutter, or a combination thereof.In some examples, the color filter array may include a red clear clear clear (RCCC) color filter array, a red clear clear blue (RCCB) color filter array, a red blue green clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor color filter array (RGGB), a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear-pixel cameras, such as cameras with an RCCC, an RCCB, and / or an RBGC color filter array, may be used to increase light sensitivity.

[0147] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multifunction mono camera can be installed to provide features including lane departure warning, traffic sign assist, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0148] One or more of the cameras can be mounted in a bracket, e.g., a specially designed (3D-printed) mount to eliminate stray light and reflections from the vehicle's interior (e.g., dashboard reflections reflected in the windshield) that could interfere with the camera's image data acquisition. Regarding the bracket for exterior mirrors, the exterior mirrors can be custom 3D-printed so that the camera mounting plate is adapted to the shape of the exterior mirror. In some examples, the one or more cameras can be integrated into the exterior mirror. For side-mounted cameras, the one or more cameras can also be integrated into the four pillars at each corner of the cabin.

[0149] Cameras with a field of view that includes portions of the environment in front of the vehicle 900 (e.g., forward-facing cameras) can be used for the surrounding view to help identify forward paths and obstacles, and to provide information critical to establishing an occupancy grid and / or determining preferred vehicle paths with the assistance of one or more controllers 936 and / or control SoCs. Forward-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS features and systems that include lane departure warnings (LDW), autonomous cruise control (ACC), and / or other features such as traffic sign detection.

[0150] A variety of cameras can be used in a forward-facing configuration, including, for example, a monocular camera platform that includes a complementary metal oxide semiconductor (CMOS) color imager. Another example is the wide-angle cameras 970, which can be used to capture objects that enter the field of view from the periphery (e.g., pedestrians, crossing vehicles, or bicycles). Although Fig. 9B illustrates only one wide-angle camera, any number (including zero) of wide-angle cameras 970 may be present on the vehicle 900. Furthermore, any number of long-range cameras 998 (e.g., a long-range stereo camera pair) may be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The one or more long-range cameras 998 may also be used for object detection and classification, as well as basic object tracking.

[0151] Any number of stereo cameras 968 may also be included in a forward-facing configuration. In at least one embodiment, one or more of the stereo cameras 968 may include an integrated control unit comprising a scalable processing unit that can provide a programmable logic ("FPGA") and a multi-core microprocessor with an integrated controller area network ("CAN") or Ethernet interface on a single chip. Such a unit can be used to create a 3D map of the vehicle's surroundings that includes a distance estimate for all points in the image. One or more alternative stereo cameras 968 may include a compact stereo vision sensor that can include two camera lenses (one each on the left and right) and an image processing chip that can measure the distance between the vehicle and the target object and process the generated information (e.g.,metadata) to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 968 may be used in addition to or alternatively to those described here.

[0152] Cameras with a field of view that includes portions of the environment to the side of the vehicle 900 (e.g., side cameras) may be used for the environment view and provide information used to create and update the occupancy grid and to generate collision warnings in the event of a side impact. For example, the one or more environment cameras 974 (e.g., four environment cameras 974, as in Fig. 9B) may be positioned on the vehicle 900. The one or more surround cameras 974 may include one or more wide-angle cameras 970, one or more fisheye cameras, one or more 360-degree cameras, and / or the like. For example, four fisheye cameras may be mounted on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may utilize three surround cameras 974 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround camera.

[0153] Cameras with a field of view that includes portions of the environment behind the vehicle 900 (e.g., rearview cameras) may be used for parking assistance, surround view, rear impact warnings, and occupancy grid creation and updating. A variety of cameras may be used, including, but not limited to, cameras that are also suitable as one or more forward-facing cameras (e.g., one or more long-range and / or medium-range cameras 998, one or more stereo cameras 968, one or more infrared cameras 972, etc.), as described herein.

[0154] Fig. 9C is a block diagram of an example system architecture for the example autonomous vehicle 900 of Fig. 9A, according to some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are set forth as examples only. Other arrangements and elements (e.g., engines, interfaces, functions, arrangements, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions performed by entities described herein may be performed by hardware, firmware, and / or software. Various functions may be performed, for example, by a processor executing instructions stored in memory.

[0155] Each of the vehicle’s components, features and systems 900 in Fig. 9C is illustrated as being connected via bus 902. Bus 902 may include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). A CAN may be a network within vehicle 900 that serves to support the control of various features and functions of vehicle 900, such as the application of brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus may be read to determine steering wheel angle, vehicle speed, engine speed (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0156] Although bus 902 is described herein as a CAN bus, this is not intended to be a limitation. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to the CAN bus. Furthermore, while a single wire is used to represent bus 902, this is not intended to be a limitation. For example, there may be any number of buses 902, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using a different protocol. In some examples, two or more buses 902 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 902 may be used for collision avoidance functionality and a second bus 902 may be used for actuation control.In each example, each bus 902 may communicate with one of the components of the vehicle 900, and two or more buses 902 may communicate with the same components. In some examples, each SoC 904, each controller 936, and / or each computer within the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle 900) and may be connected to a common bus, such as the CAN bus.

[0157] The vehicle 900 may include one or more controllers 936 as described herein with reference to Fig. 9A. The one or more controllers 936 may be used for a variety of functions. The one or more controllers 936 may be coupled to any of the various other components and systems of the vehicle 900 and may be used for control of the vehicle 900, artificial intelligence of the vehicle 900, infotainment for the vehicle 900, and / or the like.

[0158] The vehicle 900 may include one or more systems on a chip (SoC) 904. The SoC 904 may include one or more CPUs 906, one or more GPUs 908, one or more processors 910, one or more caches 912, one or more accelerators 914, one or more data memories 916, and / or other components and features not illustrated. The one or more SoCs 904 may be used to control the vehicle 900 in a variety of platforms and systems. For example, the one or more SoCs 904 may be combined in a system (e.g., the system of the vehicle 900) with an HD card 922 that may be accessed via a network interface 922 from one or more servers (e.g., the one or more servers 978 of Fig. 9D) Receive map refreshes and / or updates.

[0159] The one or more CPUs 906 may include a CPU cluster or CPU complex (alternatively referred to herein as a "CCPLEX"). The one or more CPUs 906 may include multiple cores and / or L2 caches. For example, in some embodiments, the one or more CPUs 906 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the one or more CPUs 906 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The one or more CPUs 906 (e.g., the CCPLEX) may be configured to support concurrent operation of clusters, such that any combination of the clusters of the one or more CPUs 906 may be active at any given time.

[0160] The one or more CPUs 906 may implement power management features that include one or more of the following: individual hardware blocks may be automatically clocked when idle to conserve dynamic power; each core clock may be controlled when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power controlled; each core cluster may be independently clock controlled if all cores are clock controlled or power controlled; and / or each core cluster may be independently power controlled if all cores are power controlled. The one or more CPUs 906 may further implement an enhanced power state management algorithm in which allowable power states and expected wake-up times are established, and the hardware / microcode determines the best power state to enter for the core, cluster, and CCPLEX.The processing cores can support simplified sequences for inputting the energy state into the software, offloading the work to the microcode.

[0161] The one or more GPUs 908 may include an integrated GPU (alternatively referred to herein as an "iGPU"). The one or more GPUs 908 may be programmable and may be efficient for parallel workloads. The one or more GPUs 908 may, in some examples, utilize an extended Tensor instruction set. The one or more GPUs 908 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB of memory capacity), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512 KB of memory capacity). In some embodiments, the one or more GPUs 908 may include at least eight streaming microprocessors. The one or more GPUs 908 may utilize one or more application programming interfaces (APIs) for computations.In addition, the one or more GPUs 908 may use one or more parallel computing platforms and / or programming models (e.g., CUDA from NVIDIA).

[0162] The one or more GPUs 908 may be power-optimized for best performance in automotive and embedded use cases. For example, the one or more GPUs 908 may be fabricated on a fin field-effect transistor (FinFET). However, this is not a limitation, and the one or more GPUs 908 may also be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor may include a number of mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block can be assigned 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA TENSOR COREs with mixed precision for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64 KB register file.In addition, the streaming microprocessors can include independent parallel integer and floating-point datapaths to enable efficient execution of workloads with a mix of computations and addressing calculations. The streaming microprocessors can include independent thread scheduling to enable fine-grained synchronization and cooperation between parallel threads. The streaming microprocessors can include a combined L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0163] The one or more GPUs 908 may include high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem to provide, in some examples, a peak memory bandwidth of approximately 900 GB / second. In some examples, synchronous graphics random-access memory (SGRAM), such as double data rate type five (GDDR5), may be used in addition to or as an alternative to the HBM memory.

[0164] The one or more GPUs 908 may include unified memory technology that includes access counters to enable more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for processor-shared memory areas. In some examples, Address Translation Services (ATS) support may be used to allow the one or more GPUs 908 to directly access the page tables of the one or more CPUs 906. In such examples, if the Memory Management Unit (MMU) of the one or more GPUs 908 fails, an address translation request may be sent to the one or more CPUs 906.In response, the one or more CPUs 906 may look up the virtual-to-physical mapping for the address in their page tables and send the translation back to the one or more GPUs 908. Thus, the unified memory technology may enable a single unified virtual address space for the memory of both the one or more CPUs 906 and the one or more GPUs 908, thereby simplifying the programming of the one or more GPUs 908 and the porting of applications to the one or more GPUs 908.

[0165] Additionally, the one or more GPUs 908 may include an access counter that can track the frequency of access by the one or more GPUs 908 to memory of other processors. The access counter can help move memory pages to the physical memory of the processor that accesses the pages most frequently.

[0166] The one or more SoCs 904 may include any number of caches 912, including those described herein. For example, the one or more caches 912 may include an L3 cache available to both the one or more CPUs 906 and the one or more GPUs 908 (e.g., connected to both the one or more CPUs 906 and the one or more GPUs 908). The one or more caches 912 may include a write-back cache that may track line states, e.g., by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may also be used.

[0167] The one or more SoCs 904 may include one or more arithmetic logic units (ALUs) that may be utilized in performing processing related to any of the various tasks or operations of the vehicle 900, such as processing DNNs. Additionally, the one or more SoCs 904 may include one or more floating-point units (FPUs)—or other mathematical co-processors or numerical co-processors—for performing mathematical operations within the system. For example, the one or more SoCs 904 may include one or more FPUs integrated as execution units within one or more CPUs 906 and / or one or more GPUs 908.

[0168] The one or more SoCs 904 may include one or more accelerators 914 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the one or more SoCs 904 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the one or more GPUs 908 and offload some of the tasks of the one or more GPUs 908 (e.g., to free up more cycles of the one or more GPUs 908 to perform other tasks). For example, the one or more accelerators 914 may be configured for targeted workloads (e.g.,Perception, convolutional neural networks (CNNs), etc.) that are robust enough to be suitable for acceleration can be used. The term "CNN" as used here can include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0169] The one or more accelerators 914 (e.g., the hardware acceleration cluster) may include a deep learning accelerator (DLA). The one or more DLAs may include one or more tensor processing units (TPUs) configured to provide an additional tens of trillion operations per second for deep learning applications and inferencing. The TPUs may be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). The one or more DLAs may also be optimized for a specific set of neural network types and floating-point operations, as well as for inferencing. The design of the one or more DLAs may provide more performance per millimeter than a general-purpose GPU, far exceeding the performance of a CPU.The one or more TPUs can perform multiple functions, including a single-instance convolution function that supports, for example, INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0170] The one or more DLAs can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for a variety of functions, including, for example and without limitation: a CNN for object identification and detection using data from camera sensors; a CNN for distance estimation using data from camera sensors; a CNN for emergency vehicle detection and identification using data from microphones; a CNN for facial recognition and vehicle owner identification using data from camera sensors; and / or a CNN for safety and / or security events.

[0171] The one or more DLAs can perform any function of the one or more GPUs 908, and by using an inference accelerator, a developer can, for example, dedicate either the one or more DLAs or the one or more GPUs 908 to each function. For example, the developer can focus the processing of CNNs and floating-point operations on the one or more DLAs and leave other functions to the one or more GPUs 908 and / or other accelerators 914.

[0172] The one or more accelerators 914 (e.g., the hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may also be referred to herein as a computer vision accelerator. The one or more PVAs may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The one or more PVAs may provide a balance between performance and flexibility. For example, and without limitation, each PVA may include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA) cores, and / or any number of vector processors.

[0173] The RISC cores may interact with image sensors (e.g., the image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of the RISC cores may include any amount of memory. The RISC cores may use any number of protocols, depending on the embodiment. In some examples, the RISC cores may execute a real-time operating system (RTOS). The RISC cores may be implemented with one or more integrated circuits, application-specific integrated circuits (ASICs), and / or memory devices. The RISC cores may include, for example, an instruction cache and / or tightly coupled RAM.

[0174] The DMA may enable components of the PVA(s) to access the system's memory independently of the one or more CPUs 906. The DMA may support any number of features designed to optimize the PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In some examples, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0175] The vector processors may be programmable processors that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing functions. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or memory (e.g., VMEM).A VPU core can contain a digital signal processor, such as a single instruction multiple data (SIMD) and very long instruction word (VLIW) digital signal processor. Combining SIMD and VLIW can increase throughput and speed.

[0176] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. Therefore, in some examples, each of the vector processors may be configured to operate independently of the other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, the multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may concurrently execute different computer vision algorithms on the same image, or even execute different algorithms on consecutive images or portions of an image.Among other things, any number of PVAs can be included in the hardware acceleration cluster, and any number of vector processors can be included in each of the PVAs. Furthermore, the one or more PVAs can contain additional memory for error correcting code (ECC) to increase the overall security of the system.

[0177] The one or more accelerators 914 (e.g., the hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM to the one or more accelerators 914. In some examples, the on-chip memory may include at least 4 MB of SRAM, consisting of, for example, and without limitation, eight field-configurable memory blocks accessible by both the PVA and the DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and the DLA may access the memory through a backbone that provides high-speed access to the memory to the PVA and the DLA.The backbone may include an on-chip computer vision network connecting the PVA and DLA to the memory (e.g., using the APB).

[0178] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA are delivering ready and valid signals before transmitting control signals / addresses / data. Such an interface can provide separate phases and channels for transmitting control signals / addresses / data, as well as bursty communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508, although other standards and protocols can also be used.

[0179] In some examples, the one or more SoCs 904 may include a real-time ray tracing hardware accelerator, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the positions and extents of objects (e.g., within a world model) to generate real-time visualization simulations, for radar signal interpretation, for sound propagation synthesis and / or analysis, for simulation of sonar systems, for general wave propagation simulation, for comparison with lidar data for localization, and / or for other functions, and / or for other purposes. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more operations related to ray tracing.

[0180] The one or more accelerators 914 (e.g., the hardware accelerator cluster) have a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for critical processing steps in ADAS and autonomous vehicles. The capabilities of the PVA are well suited to algorithmic areas that require predictable processing with low power and low latency. In other words, the PVA is well suited for semi-dense or dense regular computations, even on small datasets, that require predictable runtimes with low latency and low power. In the context of autonomous vehicle platforms, the PVAs are therefore designed to execute classical computer vision algorithms because they are efficient at object detection and operate on integer mathematics.

[0181] According to one embodiment of the technology, the PVA is used, for example, to perform computer stereovision. In some examples, a semi-global matching-based algorithm may be used, although this is not intended as a limitation. Many applications for Level 3-5 autonomous driving require on-the-fly motion estimation or stereo matching (e.g., structure from motion, pedestrian detection, lane detection, etc.). The PVA can perform a computer stereovision function on inputs from two monocular cameras.

[0182] In some examples, the PVA can be used to perform dense optical flow, such as processing raw radar data (e.g., using a 4D Fast Fourier Transform) to provide processed radar. In other examples, the PVA is used for time-of-flight depth processing, such as processing raw time-of-flight data to provide processed time-of-flight data.

[0183] The DLA can be used to power any type of network to improve control and driving safety; for example, this includes a neural network that outputs a confidence measure for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative "weight" to each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only those detections that exceed the threshold as true positives.In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most certain detections should be considered as triggers for AEB. The DLA may employ a neural network to regress the confidence value. The neural network may take as input at least a subset of parameters, such as, but not limited to, the bounding box dimensions, the ground plane estimate obtained (e.g., from another subsystem), the output of the inertial measurement unit (IMU) sensor 966 correlated with the orientation of the vehicle 900, distance, 3D position estimates of the object obtained by the neural network and / or other sensors (e.g., one or more LIDAR sensors 964 or one or more RADAR sensors 960).

[0184] The one or more SoCs 904 may include one or more data stores 916 (e.g., memory). The one or more data stores 916 may be on-chip memory on the one or more SoCs 904 in which neural networks to be executed on the GPU and / or the DLA may be stored. In some examples, the one or more data stores 916 may be large enough to store multiple instances of neural networks for redundancy and safety. The one or more data stores 912 may include one or more L2 or L3 caches 912. The reference to the one or more data stores 916 may include a reference to the memory associated with the PVA, the DLA, and / or one or more other accelerators 914, as described herein.

[0185] The one or more SoCs 904 may include one or more processors 910 (e.g., embedded processors). The one or more processors 910 may include a boot and power management processor, which may be a dedicated processor and subsystem to handle boot power and management functions and associated security enforcement. The boot and power management processor may be part of the boot sequence of the one or more SoCs 904 and may provide runtime power management services. The boot and power management processor may provide clock and voltage programming, assisting with system transitions to a low-power state, managing the thermals and temperature sensors of the one or more SoCs 904, and / or managing the one or more SoCs 904 power states.Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the one or more SoCs 904 may use the ring oscillators to sense the temperatures of the one or more CPUs 906, the one or more GPUs 908, and / or the one or more accelerators 914. If it is determined that the temperatures exceed a threshold, the boot and power management processor may enter a temperature fault routine and place the one or more SoCs 904 into a lower power state and / or place the vehicle 900 into a chauffeur-to-safe-stop mode (e.g., bring the vehicle 900 to a safe stop).

[0186] The one or more processors 910 may also include a set of embedded processors that can serve as an audio processing engine. The audio processing engine may be an audio subsystem that enables full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor with dedicated RAM.

[0187] The one or more processors 910 may also include an always-on processor engine that can provide the necessary hardware functions to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0188] The one or more processors 910 may also include a safety cluster engine containing a dedicated processor subsystem for safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, an interrupt controller, etc.), and / or routing logic. In a safety mode, the two or more cores may operate in a lockstep mode, functioning as a single core with comparison logic that captures any differences between their operations.

[0189] The one or more processors 910 may also include a real-time camera engine, which may include a dedicated processor subsystem for managing the real-time camera.

[0190] The one or more processors 910 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0191] The one or more processors 910 may include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate the final image for the player window. The video image compositor may perform lens distortion correction on the one or more wide-angle cameras 970, the one or more surround cameras 974, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensor is preferably monitored by a neural network running on another instance of the enhanced SoC and configured to detect events in the cabin and respond accordingly.An in-cabin system can perform lip reading to activate cellular service and make a call, dictate emails, change the destination, activate or change the vehicle's infotainment system and settings, or enable voice-activated web browsing. Certain functions are available to the driver only when the vehicle is operating in autonomous mode and are disabled otherwise.

[0192] The video image compositor can incorporate enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in a video, the noise reduction weights the spatial information accordingly, reducing the weight of information provided by neighboring frames. If a frame or section of a frame contains no motion, the temporal noise reduction performed by the video image compositor can use information from the previous frame to reduce noise in the current frame.

[0193] The video image compositor may also be configured to perform stereo distortion correction on the input stereo lens images. The video image compositor may also be used for user interface design when the operating system desktop is in use and the one or more GPUs 908 do not need to constantly render new surfaces. Even when the one or more GPUs 908 are powered on and actively performing 3D rendering, the video image compositor may be used to offload the one or more GPUs 908, thereby improving performance and responsiveness.

[0194] The one or more SoCs 904 may also include a Mobile Industry Processor Interface (MIPI) serial camera interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that may be used for camera and related pixel input functions. The one or more SoCs 904 may also include one or more input / output controllers, one or more of which may be software-controlled and used to receive I / O signals that are not associated with a specific role.

[0195] The one or more SoCs 904 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The one or more SoCs 904 may be used to process data from cameras (e.g., via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., one or more LIDAR sensors 964, one or more RADAR sensors 960, etc., which may be connected via Ethernet), data from bus 902 (e.g., speed of vehicle 900, steering wheel position, etc.), data from one or more GNSS sensors 958 (e.g., connected via Ethernet or CAN bus).The one or more SoCs 904 may further include dedicated high-performance mass storage controllers, which may include their own DMA engines and which may be used to offload routine data management tasks from the one or more CPUs 906.

[0196] The one or more SoCs 904 may be an end-to-end platform with a flexible architecture spanning automation levels 3-5, thereby providing a comprehensive functional safety architecture that supports and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The one or more SoCs 904 may be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the one or more accelerators 914, in combination with the one or more CPUs 906, the one or more GPUs 908, and the one or more data memories 916, may form a fast, efficient platform for Level 3-5 autonomous vehicles.

[0197] The technology thus offers capabilities and functions that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using a high-level programming language, such as the C programming language, to execute a variety of processing algorithms on a wide variety of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, such as execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a prerequisite for in-vehicle ADAS applications and a requirement for practical Level 3-5 autonomous vehicles.

[0198] In contrast to conventional systems, the technology described herein enables the simultaneous and / or sequential execution of multiple neural networks and the combination of the results to enable Level 3-5 autonomous driving functionality by providing a CPU complex, a GPU complex, and a hardware acceleration cluster. For example, a CNN running on the DLA or dGPU (e.g., the one or more GPUs 920) may include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing semantic understanding of the sign, and passing this semantic understanding to the path planning modules running on the CPU complex.

[0199] Another example is that multiple neural networks can run simultaneously, as required for driving at Level 3, 4, or 5. For example, a warning sign reading "Caution: Flashing lights indicate black ice," along with an electric light, can be interpreted independently or jointly by multiple neural networks. The sign itself can be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing lights indicate black ice" can be interpreted by a second deployed neural network, which informs the vehicle's path-planning software (preferably running on the CPU complex) that if flashing lights are detected, black ice is present.The turn signal can be identified across multiple images by a third neural network, which informs the vehicle's path planning software of the presence (or absence) of turn signals. All three neural networks can run simultaneously, e.g., within the DLA and / or on one or more GPUs 908.

[0200] In some examples, a facial recognition and vehicle owner identification CNN may use data from camera sensors to identify the presence of an authorized driver and / or owner of the vehicle 900. The always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and to disable the vehicle in security mode when the owner exits the vehicle. In this way, the one or more SoCs 904 provide security against theft and / or carjacking.

[0201] In another example, a CNN for detecting and identifying emergency vehicles may use data from microphones 996 to detect and identify emergency vehicle sirens. Unlike conventional systems that use general classifiers to detect sirens and manually extract features, the one or more SoCs 904 use the CNN to classify environmental and urban sounds, as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to detect the relative approach speed of the emergency vehicle (e.g., by using the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by one or more GNSS sensors 958.For example, if the CNN is operating in Europe, it will attempt to detect European sirens, and if it is operating in the United States, the CNN will attempt to identify only North American sirens. Once an emergency vehicle is detected, a controller can be used to execute an emergency vehicle safety routine, slow the vehicle, pull over to the side of the road, park the vehicle, and / or idle the vehicle, using ultrasonic sensors 962, until the one or more emergency vehicles pass by.

[0202] The vehicle may include one or more CPUs 918 (e.g., one or more discrete CPUs or one or more dCPUs) that may be coupled to the one or more SoCs 904 via a high-speed connection (e.g., PCIe). The CPUs 918 may include, for example, an x86 processor. The CPUs 918 may be used, for example, to perform a variety of functions, including reconciling potentially inconsistent results between ADAS sensors and the one or more SoCs 904 and / or monitoring the status and health of the one or more controllers 936 and / or the infotainment SoC 930.

[0203] The vehicle 900 may include one or more GPUs 920 (e.g., one or more discrete GPUs or one or more dGPUs) that may be coupled to the one or more SoCs 904 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The one or more GPUs 920 may provide additional artificial intelligence capabilities, e.g., by executing redundant and / or distinct neural networks, and may be used to train and / or update neural networks based on inputs (e.g., sensor data) from sensors of the vehicle 900.

[0204] The vehicle 900 may further include the network interface 922, which may include one or more wireless antennas 926 (e.g., one or more wireless antennas for various communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 922 may be used to enable a wireless connection over the internet to the cloud (e.g., to the one or more servers 978 and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct connection between the two vehicles and / or an indirect connection may be established (e.g., via networks and the internet). Direct connections may be established via vehicle-to-vehicle communication.Vehicle-to-vehicle communication may provide vehicle 900 with information about vehicles in the vicinity of vehicle 900 (e.g., vehicles in front of, beside, and / or behind vehicle 900). This functionality may be part of a cooperative adaptive cruise control feature of vehicle 900.

[0205] The network interface 922 may include an SoC that provides modulation and demodulation functions and enables the one and more controllers 936 to communicate over wireless networks. The network interface 922 may include a radio frequency front end for upconverting from baseband to radio frequency and downconverting from radio frequency to baseband. The frequency conversions may be performed using known methods and / or super-heterodyne methods. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0206] The vehicle 900 may further include one or more data stores 928, which may be located off-chip (e.g., outside the SoCs 904). The one or more data stores 928 may include one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard drives, and / or other components and / or devices capable of storing at least one bit of data.

[0207] The vehicle 900 may further include one or more GNSS sensors 958. The one or more GNSS sensors 958 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) assist in mapping, sensing, occupancy grid creation, and / or path planning. Any number of GNSS sensors 958 may be used, including, for example, and without limitation, a GPS using a USB port with an Ethernet-to-serial (RS-232) bridge.

[0208] The vehicle 900 may further include one or more RADAR sensors 960. The one or more RADAR sensors 960 may be used by the vehicle 900 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The functional safety level of the RADAR may be ASIL B. The one or more RADAR sensors 960 may utilize the CAN and / or bus 902 (e.g., to transmit the data generated by the one or more RADAR sensors 960) for control and access to object tracking data, with access to the raw data occurring over Ethernet in some examples. A variety of RADAR sensor types may be used. The one or more RADAR sensors 960 may be suitable for, for example, front-, rear-, and side-facing RADAR, without limitation. In some examples, one or more Pulse Doppler RADAR sensors are used.

[0209] The one or more RADAR sensors 960 may include various configurations, such as long range with a narrow field of view, short range with a wide field of view, short range side coverage, etc. In some examples, long range RADAR may be used for the adaptive cruise control function. Long range RADAR systems may provide a wide field of view realized through two or more independent scans, such as at a range of 250 m. The one or more RADAR sensors 960 may assist in distinguishing between static and moving objects and may be used by ADAS systems for emergency braking and forward collision warning. Long range RADAR sensors may include a monostatic multi-modal RADAR with multiple (e.g., six or more) fixed RADAR antennas and a high-speed CAN and FlexRay interface.In a six-antenna example, the middle four antennas can create a focused beam pattern designed to detect the surroundings of vehicle 900 at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can expand the field of view so that vehicles entering or exiting the lane of vehicle 900 can be quickly detected.

[0210] Medium-range radar systems, for example, can have a range of up to 1260 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1250 degrees (rear). Short-range radar systems can include, among other features, radar sensors designed for installation at both ends of the rear bumper. When installed at both ends of the rear bumper, such a radar sensor system can generate two beams that continuously monitor the blind spot area to the rear and to the side of the vehicle.

[0211] Short-range radar systems can be used in an ADAS system to monitor blind spots and / or assist with lane change.

[0212] The vehicle 900 may also include one or more ultrasonic sensors 962. The one or more ultrasonic sensors 962, which may be mounted on the front, rear, and / or sides of the vehicle 900, may be used for parking assistance and / or for creating and updating an occupancy grid. A plurality of ultrasonic sensors 962 may be used, and different ultrasonic sensors 962 may be used for different detection ranges (e.g., 2.5 m, 4 m). The one or more ultrasonic sensors 962 may operate at functional safety levels of ASIL B.

[0213] The vehicle 900 may include one or more LIDAR sensors 964. The one or more LIDAR sensors 964 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The one or more LIDAR sensors 964 may conform to ASIL B functional safety level. In some examples, the vehicle 900 may include multiple LIDAR sensors 964 (e.g., two, four, six, etc.) that may use Ethernet (e.g., to deliver data to a Gigabit Ethernet switch).

[0214] In some examples, the one or more LIDAR sensors 964 may be capable of providing a list of objects and their distances for a 360-degree field of view. For example, commercially available LIDAR sensors 964 may have an advertised range of approximately 1200 m, with an accuracy of 2 cm to 3 cm, and with support for a 1200 Mbps Ethernet connection. In some examples, one or more non-prominent LIDAR sensors 964 may be used. In such examples, the one or more LIDAR sensors 964 may be implemented as a small device that may be embedded in the front, rear, sides, and / or corners of the vehicle 900. In such examples, the one or more LIDAR sensors 964 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees, with a range of 200 m, even for objects with low reflectivity.The one or more front-mounted LIDAR sensors 964 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0215] In some examples, LIDAR technologies such as 3D flash LIDAR may also be used. 3D flash LIDAR uses a laser flash as the transmission source to illuminate the vehicle's surroundings up to approximately 200 m. A flash LIDAR unit contains a receptor that records the time of flight of the laser pulse and the reflected light at each pixel, which in turn corresponds to the distance between the vehicle and the objects. Flash LIDAR can enable highly accurate and distortion-free images of the surroundings to be created with each laser flash. In some examples, four flash LIDAR sensors may be deployed, one on each side of the vehicle. Available 3D flash LIDAR systems include a solid-state 3D focal plane array LIDAR camera that contains no moving parts other than a fan (e.g., a non-scanning LIDAR device).The flash lidar device can use a 5-nanosecond pulse of a Class I (eye-safe) laser per frame and collect the reflected laser light as 3D range point clouds and co-registered intensity data. By using flash lidar, and because flash lidar is a solid-state device with no moving parts, the one or more lidar sensors 964 can be less susceptible to motion blur, vibration, and / or shock.

[0216] The vehicle may also include one or more IMU sensors 966. The one or more IMU sensors 966 may, in some examples, be located at the center of the rear axle of the vehicle 900. The one or more IMU sensors 966 may, for example and without limitation, include one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other types of sensors. In some examples, such as in six-axis applications, the one or more IMU sensors 966 may include accelerometers and gyroscopes, while in nine-axis applications, the one or more IMU sensors 966 may include accelerometers, gyroscopes, and magnetometers.

[0217] In some embodiments, the one or more IMU sensors 966 may be implemented as a miniaturized, high-performance GPS-aided inertial navigation system (GPS / INS) that combines microelectromechanical system (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and attitude. Thus, in some examples, the one or more IMU sensors 966 may enable the vehicle 900 to estimate heading without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS with the one or more IMU sensors 966. In some examples, the one or more IMU sensors 966 and the one or more GNSS sensors 958 may be combined into a single integrated unit.

[0218] The vehicle may include one or more microphones 996 mounted in and / or around the vehicle 900. The one or more microphones 996 may be used, among other things, to detect and identify emergency vehicles.

[0219] The vehicle may further include any number of camera types, including one or more stereo cameras 968, one or more wide-angle cameras 970, one or more infrared cameras 972, one or more surround cameras 974, one or more long-range and / or medium-range cameras 998, and / or other camera types. The cameras may be used to capture image data around the entire periphery of the vehicle 900. The types of cameras used depend on the embodiments and requirements for the vehicle 900, and any combination of camera types may be used to provide the necessary coverage around the vehicle 900. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or a different number of cameras.The cameras may support, by way of example and without limitation, Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the one or more cameras is referred to herein with reference to . Fig. 9A and Fig. 9B is described in more detail.

[0220] The vehicle 900 may further include one or more vibration sensors 942. The one or more vibration sensors 942 may measure vibrations from components of the vehicle, such as the one or more axles. For example, changes in vibrations may indicate a change in the road surface. In another example, when two or more vibration sensors 942 are used, the differences between the vibrations may be used to determine friction or slippage on the road surface (e.g., when the difference in vibration is between a driven axle and a free-spinning axle).

[0221] The vehicle 900 may include an ADAS system 938. The ADAS system 938 may include an SoC in some examples. The ADAS system 938 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (CWS), lane centering (LC), and / or other features and functions.

[0222] The ACC systems may use one or more RADAR sensors 960, one or more LIDAR sensors 964, and / or one or more cameras. The ACC systems may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle 900 and automatically adjusts the vehicle speed to maintain a safe distance from preceding vehicles. Lateral ACC performs follow-through and advises the vehicle 900 to change lanes if necessary. Lateral ACC is associated with other ADAS applications, such as LCA and CWS.

[0223] The CACC utilizes information from other vehicles, which may be received via the network interface 922 and / or the one or more wireless antennas 926 from other vehicles over a wireless connection or indirectly via a network connection (e.g., over the Internet). Direct connections may be provided via a vehicle-to-vehicle (V2V) communication link, while indirect connections may be an infrastructure-to-vehicle (I2V) communication link. In general, the V2V communication concept provides information about the immediately preceding vehicles (e.g., vehicles immediately in front of and in the same lane as vehicle 900), while the I2V communication concept provides information about traffic further ahead. CACC systems may include both I2V and V2V information sources.Given the information about the vehicles ahead of vehicle 900, CACC can be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

[0224] FCW systems are designed to warn the driver of a hazard so they can take corrective action. FCW systems utilize a forward-facing camera and / or one or more radar sensors 960 coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component. FCW systems can provide a warning, such as a sound, a visual warning, a vibration, and / or a rapid braking pulse.

[0225] AEB systems detect an impending forward collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. AEB systems may utilize one or more forward-facing cameras and / or one or more radar sensors 960 coupled with a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid the collision; if the driver does not take corrective action, the AEB system can automatically apply the brakes to prevent or at least mitigate the effects of the predicted collision. AEB systems may incorporate techniques such as dynamic brake support and / or impending crash braking.

[0226] LDW systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 900 crosses lane markings. An LDW system will not activate if the driver indicates an intentional lane departure by activating a turn signal. LDW systems may utilize forward-facing cameras coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component.

[0227] LKA systems are a variant of LDW systems. LKA systems provide steering inputs or braking to correct the vehicle 900 when the vehicle 900 begins to depart from its lane.

[0228] BSW systems detect and warn the driver of vehicles in the car's blind spot. BSW systems may provide a visual, audible, and / or tactile warning signal to indicate that merging into or changing lanes is unsafe. The system may provide an additional warning when the driver activates a turn signal. BSW systems may utilize one or more rear-facing cameras and / or radar sensors 960 coupled with a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component.

[0229] RCTW systems can provide visual, audible, and / or tactile notification when an object outside the range of the rearview camera is detected when the vehicle 900 is reversing. Some RCTW systems include AEB to ensure the vehicle brakes are applied to avoid a crash. RCTW systems can utilize one or more rear-facing RADAR sensors 960 coupled with a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to feedback to the driver, such as a display, speaker, and / or vibrating component.

[0230] Conventional ADAS systems can produce false positives, which can be annoying and distracting for the driver, but are typically not catastrophic because ADAS systems warn the driver and give them the opportunity to decide whether a safety issue truly exists and act accordingly. However, in an autonomous vehicle 900, in the event of conflicting results, the vehicle 900 must decide for itself whether to consider the result of a primary computer or a secondary computer (e.g., a first controller 936 or a second controller 936). In some embodiments, the ADAS system 938 may, for example, be a backup and / or secondary computer that provides perception information to a rationality module of the backup computer.The backup computer rationality monitor can run redundant, diverse software on hardware components to detect errors in perception and dynamic driving tasks. The outputs of the ADAS system 938 can be provided to a supervising MCU. If the outputs of the primary computer and the secondary computer conflict, the supervising MCU must determine how to resolve the conflict to ensure safe operation.

[0231] In some examples, the primary computer may be configured to provide the monitoring MCU with a confidence value indicating the primary computer's confidence in the chosen outcome. If the confidence value exceeds a threshold, the monitoring MCU may follow the primary computer's instruction regardless of whether the secondary computer provides a conflicting or inconsistent result. If the confidence value does not meet the threshold and the primary and secondary computers indicate different results (e.g., a conflict), the monitoring MCU may arbitrate between the computers to determine the appropriate outcome.

[0232] The monitoring MCU may be configured to run one or more neural networks trained and configured to determine the conditions under which the secondary computer triggers false alarms based on the output from the primary and secondary computers. This allows the one or more neural networks in the monitoring MCU to learn when the output of the secondary computer can and cannot be trusted. For example, if the secondary computer is a radar-based FCW system, a neural network in the monitoring MCU can learn when the FCW system identifies metallic objects that are not actually hazardous, such as a drain grate or manhole cover, which triggers an alarm.Similarly, if the secondary computer is a camera-based LDW system, a neural network in the monitoring MCU can learn to override the LDW system when cyclists or pedestrians are present and lane departure is indeed the safest maneuver. In embodiments including one or more neural networks running on the monitoring MCU, the monitoring MCU can include at least one DLA or GPU suitable for executing the one or more neural networks with associated memory. In preferred embodiments, the monitoring MCU can comprise and / or be included as a component of the one or more SoCs 904.

[0233] In other examples, the ADAS system 938 may include a secondary computer that executes the ADAS functionality according to classical computer vision rules. Thus, the secondary computer may use classical computer vision rules (if-then), and the presence of one or more neural networks in the supervising MCU may improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially against errors caused by software (or software-hardware interfaces).For example, if a software bug or error occurs in the software on the primary computer and the non-identical software code on the secondary computer produces the same overall result, the monitoring MCU can have greater confidence that the overall result is correct and the bug in the software or hardware on the primary computer does not cause a significant error.

[0234] In some examples, the output of the ADAS system 938 may be fed into the perception block of the primary computer and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 938 displays a forward collision warning due to an object immediately in front of the vehicle, the perception block may use this information in identifying objects. In other examples, the secondary computer may have its own neural network trained to reduce the risk of false positives, as described herein.

[0235] The vehicle 900 may further include the infotainment SoC 930 (e.g., an in-vehicle infotainment (IVI) system). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 930 may include a combination of hardware and software that can be used to provide the vehicle 900 with audio (e.g., music, a personal digital assistant, navigation directions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, rear parking assist, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fuel level, oil level, door open / close, air filter information, etc.).The infotainment SoC 930 may include, for example, radios, record players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free calling, a head-up display (HUD), an HMI display 934, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. The infotainment SoC 930 may further be used to provide information (e.g., visual and / or audible) to one or more users of the vehicle, such as information from the ADAS system 938, autonomous driving information such as planned vehicle maneuvers, road layouts, environmental information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0236] The infotainment SoC 930 may include GPU functionality. The infotainment SoC 930 may communicate with other devices, systems, and / or components of the vehicle 900 via the bus 902 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 930 may be coupled to a supervisory MCU so that the GPU of the infotainment system can perform some self-driving functions if the one or more primary controllers 936 (e.g., the primary and / or backup computers of the vehicle 900) fail. In such an example, the infotainment SoC 930 may place the vehicle 900 into a chauffeur-to-safe-stop mode, as described herein.

[0237] The vehicle 900 may further include an instrument cluster 932 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 932 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 932 may include a number of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, shift position indicator, seat belt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag system (SRS) information, lighting controls, safety system controls, navigation information, etc. In some examples, information from the infotainment SoC 930 and the instrument cluster 932 may be displayed and / or shared. In other words, the instrument cluster 932 may be included as part of the infotainment SoC 930, or vice versa.

[0238] Fig. 9D is a system diagram for communication between the one or more cloud-based servers and the example autonomous vehicle 900 of Fig. 9A, according to some embodiments of the present disclosure. The system 976 may include the one or more servers 978, the one or more networks 990, and the vehicles, including the vehicle 900. The server(s) 978 may include a plurality of GPUs 984(A)-984(H) (collectively referred to herein as GPUs 984), PCIe switches 982(A)-982(H) (collectively referred to herein as PCIe switches 982), and / or CPUs 980(A)-980(B) (collectively referred to herein as CPUs 980). The GPUs 984, the CPUs 980, and the PCIe switches may be interconnected with high-speed interconnects, such as, without limitation, the NVIDIA-developed NVLink interfaces 988 and / or PCIe interconnects 986. In some examples, the GPUs 984 are connected via NVLink and / or NVSwitch SoC, and the GPUs 984 and the PCIe switches 982 are connected via PCIe links.Although eight GPUs 984, two CPUs 980, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 978 may include any number of GPUs 984, CPUs 980, and / or PCIe switches. For example, the one or more servers 978 may each include eight, sixteen, thirty-two, and / or more GPUs 984.

[0239] The one or more servers 978 may receive, via the one or more networks 990 and from the vehicles, image data representative of images depicting unexpected or changed road conditions, such as recently commenced roadwork. The one or more servers 978 may transmit, via the one or more networks 990 and to the vehicles, neural networks 992, updated neural networks 992, and / or map information 994 containing information about traffic and road conditions. The updates to the map information 994 may include updates to the HD map 922, such as information about construction, potholes, detours, flooding, and / or other obstacles.In some examples, the neural networks 992, the updated neural networks 992, and / or the map information 994 may result from new training and / or experience represented in the data received from any number of vehicles in the environment and / or may be based on training performed in a data center (e.g., using the one or more servers 978 and / or other servers).

[0240] The one or more servers 978 may be used to train machine learning models (e.g., neural networks) based on training data. The training data may be generated by the vehicles and / or in a simulation (e.g., using a game engine). In some examples, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not tagged and / or preprocessed (e.g., if the neural network does not require supervised learning).Training may be performed using one or more classes of machine learning techniques, including, without limitation, supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including replacement dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning models are trained, the machine learning models may be used by the vehicles (e.g., transmitted to the vehicles via the one or more networks 990) and / or the machine learning models may be used by the one or more servers 978 to remotely monitor the vehicles.

[0241] In some examples, the one or more servers 978 may receive data from the vehicles and apply the data to real-time, real-time neural networks for intelligent inference. The one or more servers 978 may include deep learning supercomputers and / or dedicated AI computers powered by GPUs 984, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the one or more servers 978 may include a deep learning infrastructure using only CPU-powered data centers.

[0242] The deep learning infrastructure of the one or more servers 978 may be capable of performing rapid inference in real time and may utilize this capability to evaluate and verify the state of the processors, software, and / or associated hardware in the vehicle 900. For example, the deep learning infrastructure may receive periodic updates from the vehicle 900, such as a sequence of images and / or objects that the vehicle 900 has located in that sequence of images (e.g., via computer vision and / or other machine learning object classification techniques).The deep learning infrastructure may run its own neural network to identify the objects and compare them to the objects identified by the vehicle 900, and if the results do not match and the infrastructure concludes that the AI in the vehicle 900 is not functioning properly, the one or more servers 978 may send a signal to the vehicle 900 instructing a fail-safe computer of the vehicle 900 to take control, notify passengers, and perform a safe parking maneuver.

[0243] For inferencing, the one or more servers 978 may include GPUs 984 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-driven servers and inference accelerators may enable real-time responsiveness. In other examples, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors may be used for inferencing. EXAMPLE CALCULATION DEVICE

[0244] Fig. 10 is a block diagram of an example computing device 1000 suitable for use in implementing some embodiments of the present disclosure. The computing device 1000 may include an interconnect system 1002 that directly or indirectly couples the following devices: memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communications interface 1010, input / output (I / O) ports 1012, input / output components 1014, a power supply 1016, one or more presentation components 1018 (e.g., display(s)), and one or more logic units 1020. In at least one embodiment, the one or more computing devices 1000 may include one or more virtual machines (VMs), and / or each of the components thereof may include virtual components (e.g., virtual hardware components).As non-limiting examples, one or more of the GPUs 1008 may include one or more vGPUs, one or more of the CPUs 1006 may include one or more vCPUs, and / or one or more of the logic units 1020 may include one or more virtual logic units. Thus, a computing device 1000 may include discrete components (e.g., a full GPU associated with the computing device 1000), virtual components (e.g., a portion of a GPU associated with the computing device 1000), or a combination thereof.

[0245] Although the different blocks of Fig. 10 as being connected to wires via the interconnect system 1002, this is not intended as a limitation and is for clarity only. For example, in some embodiments, a presentation component 1018, such as a display device, may be considered an I / O component 1014 (e.g., if the display is a touchscreen). As another example, the CPUs 1006 and / or GPUs 1008 may include memory (e.g., the memory 1004 may represent a storage device in addition to the memory of the GPUs 1008, the CPUs 1006, and / or other components). In other words, the computing device of Fig. Figure 10 is merely illustrative. No distinction is made between categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, since all devices within the scope of the computing device are Fig. 10 come into consideration.

[0246] The interconnect system 1002 may represent one or more interconnects or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 1002 may include one or more bus or interconnect types, such as an Industry Standard Architecture (ISA) bus, an Extended ISA bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, and / or another type of bus or interconnect. In some embodiments, there are direct connections between components. For example, the CPU 1006 may be directly connected to the memory 1004. Further, the CPU 1006 may be directly connected to the GPU 1008.For a direct or point-to-point connection between components, interconnect system 1002 may include a PCIe link to establish the connection. In these examples, a PCI bus need not be included in computing device 1000.

[0247] Memory 1004 may include a variety of computer-readable media. The computer-readable media may be any available media accessible by the computing device 1000. The computer-readable media may include both volatile and non-volatile media, as well as removable and non-removable media. By way of example and without limitation, the computer-readable media may include computer storage media and communication media.

[0248] The computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, the memory 1004 may store computer-readable instructions (e.g., representing one or more programs) and / or one or more program elements, such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 1000.As used herein, computer storage media does not per se include signals.

[0249] The computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and may include any media for conveying information. The term "modulated data signal" may refer to a signal having one or more of its characteristics adjusted or altered to encode information in the signal. The computer storage media may include, for example, and is not limited to, wired media, such as a wired network or a direct-wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of the above should also be within the scope of computer-readable media.

[0250] The one or more CPUs 1006 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. The one or more CPUs 1006 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of concurrently executing a plurality of software threads. The one or more CPUs 1006 may include any type of processor and may include different types of processors depending on the type of computing device 1000 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).Depending on the type of computing device 1000, the processor may be, for example, an Advanced RISC Machines (ARM) processor implemented with Reduced Instruction Set Computing (RISC) or an x86 processor implemented with Complex Instruction Set Computing (CISC). Computing device 1000 may include one or more CPUs 1006, in addition to one or more microprocessors or additional coprocessors, such as math coprocessors.

[0251] In addition to or alternatively to the one or more CPUs 1006, the one or more GPUs 1008 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 1008 may be an integrated GPU (e.g., with one or more of the CPUs 1006) and / or one or more of the GPUs 1008 may be a discrete GPU. In embodiments, one or more of the GPUs 1008 may be a co-processor of one or more of the CPUs 1006. The one or more GPUs 1008 may be used by the computing device 1000 to render graphics (e.g., 3D graphics) or to perform general-purpose computations. The one or more GPUs 1008 may be used, for example, for general-purpose computing on GPUs (GPGPU).The one or more GPUs 1008 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The one or more GPUs 1008 may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the one or more CPUs 1006 received via a host interface). The one or more GPUs 1008 may include graphics memory, such as display memory, for storing pixel data or other suitable data, such as GPGPU data. The display memory may be included as part of the main memory 1004. The one or more GPUs 1008 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or connect the GPUs via a switch (e.g., using NVSwitch).When combined, each GPU can generate 1008 pixel data or GPGPU data for different sections of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can contain its own memory or share memory with other GPUs.

[0252] In addition to or alternatively to the one or more CPUs 1006 and / or the one or more GPUs 1008, the one or more logic units 1020 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, the one or more CPUs 1006, the GPUs 1008, and / or the one or more logic units 1020 may discretely or jointly execute any combination of the methods, processes, and / or portions thereof. One or more of the logic units 1020 may be part of and / or integrated with one or more of the CPUs 1006 and / or one or more of the GPUs 1008, and / or one or more of the logic units 1020 may be discrete components or otherwise external to the CPUs 1006 and / or the GPUs 1008.In embodiments, one or more of the logic units 1020 may be a co-processor of one or more of the CPUs 1006 and / or one or more of the GPUs 1008.

[0253] Examples of the one or more logic units 1020 include one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (Application-Specific Integrated Circuits, ASICs), Floating Point Units (FPUs),Input / output (I / O) elements, peripheral component interconnect (PCI) or PCI Express (PCIe) elements, and / or similar.

[0254] The communication interface 1010 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1000 to communicate with other computing devices over an electronic network, including wired and / or wireless communication. The communication interface 1010 may include components and functions that enable communication over a variety of networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.In one or more embodiments, the one or more logic units 1020 and / or the communication interface 1010 may include one or more data processing units (DPUs) to transfer data received over a network and / or via the interconnect system 1002 directly to one or more GPUs 1008 (e.g., a memory thereof).

[0255] The I / O ports 1012 may enable the computing device 1000 to be logically coupled to other devices, including the I / O components 1014, the one or more presentation components 1018, and / or other components, some of which may be built into (e.g., integrated) the computing device 1000. Illustrative I / O components 1014 include a microphone, a mouse, a keyboard, a joystick, a gamepad, a game controller, a satellite dish, a scanner, a printer, a wireless device, etc. The I / O components 1014 may provide a natural user interface (NUI) that processes air gestures, speech, or other physiological inputs generated by a user. In some cases, the inputs may be communicated to a suitable network element for further processing.An NUI may implement any combination of speech capture, stylus capture, facial capture, biometric capture, both on-screen and off-screen gesture capture, air gestures, head and eye tracking, and touch capture (as further described below) associated with a display of the computing device 1000. The computing device 1000 may include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture capture and recognition. Additionally, the computing device 1000 may include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion capture. In some examples, the output of the accelerometers or gyroscopes from the computing device 1000 may be used to present immersive augmented reality or virtual reality.

[0256] Power supply 1016 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 1016 may supply power to computing device 1000 to enable operation of the components of computing device 1000.

[0257] The one or more presentation components 1018 may include a display (e.g., a monitor, a touchscreen, a television monitor, a heads-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The one or more presentation components 1018 may receive data from other components (e.g., the one or more GPUs 1008, the one or more CPUs 1006, DPUs, etc.) and output the data (e.g., as an image, video, audio, etc.). EXEMPLARY DATA CENTER

[0258] Fig. Figure 11 illustrates an example data center 1100 that may be used in at least one embodiment of the present disclosure. Data center 1100 may include a data center infrastructure layer 1110, a framework layer 1120, a software layer 1130, and / or an application layer 1140.

[0259] As in Fig. 11, the data center infrastructure layer 1110 may include a resource orchestrator 1112, clustered computing resources 1114, and node computing resources (“node CRs”) 116(1)-1116(N), where “N” represents any positive integer. In at least one embodiment, the node CRs 1116(1)-1116(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc.In some embodiments, one or more Node CRs among Node CRs 1116(1)-1116(N) may correspond to a server having one or more of the computing resources mentioned above. Furthermore, in some embodiments, Node CRs 1116(1)-1116(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or the like, and / or one or more of Node CRs 1116(1)-1116(N) may correspond to a virtual machine (VM).

[0260] In at least one embodiment, the grouped computing resources 1114 may include separate groupings of node CRs 1116 housed in one or more racks (not shown) or in many racks in data centers in different geographic locations (also not shown). Separate groupings of node CRs 1116 within grouped computing resources 1114 may include grouped computing, network, memory, or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, multiple node CRs 1116, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads.The one or more racks may also contain any number of power modules, cooling modules, and / or network switches in any combination.

[0261] Resource orchestrator 1112 may configure or otherwise control one or more node CRs 1116(1)-1116(N) and / or clustered computing resources 1114. In at least one embodiment, resource orchestrator 1112 may include an entity for managing the software design infrastructure (SDI) for data center 1100. Resource orchestrator 1112 may include hardware, software, or a combination thereof.

[0262] In at least one embodiment, as in Fig. 11, the framework layer 1120 may include a job scheduler 1133, a configuration manager 1134, a resource manager 1136, and / or a distributed file system 1138. The framework layer 1120 may include a framework that supports the software 1132 of the software layer 1130 and / or one or more applications 1142 of the application layer 1140. The software 1132 or the one or more applications 1142 may each include web-based service software or applications such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1120 may be some type of free and open-source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter "Spark"), which may utilize a distributed file system 1138 for processing large amounts of data (e.g., "Big Data").In at least one embodiment, job scheduler 1133 may include a Spark driver to facilitate scheduling workloads supported by different layers of data center 1100. Configuration manager 1134 may be capable of configuring different layers, such as software layer 1130 and framework layer 1120, which includes Spark and distributed file system 1138, to support processing large amounts of data. Resource manager 1136 may be capable of managing clustered or grouped computing resources allocated or assigned to support distributed file system 1138 and job scheduler 1133. In at least one embodiment, the clustered or grouped computing resources may include clustered computing resource 1114 at infrastructure layer 1110 of the data center.The resource manager 1136 may coordinate with the resource orchestrator 1112 to manage these allocated or assigned computing resources.

[0263] In at least one embodiment, the software 1132 included in software layer 1130 may include software used by at least portions of node CRs 1116(1)-1116(N), clustered computing resources 1114, and / or distributed file system 1138 of framework layer 1120. One or more types of software may include, but are not limited to, web page searching software, email virus scanning software, database software, and streaming video content software.

[0264] In at least one embodiment, the applications 1142 included in the application layer 1140 may include one or more types of applications used by at least portions of the node CRs 1116(1)-1116(N), the clustered computing resources 1114, and / or the distributed file system 1138 of the framework layer 1120. One or more types of applications may include, but are not limited to, any number of genomic applications, cognitive computation, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in connection with one or more embodiments.

[0265] In at least one embodiment, one of configuration manager 1134, resource manager 1136, and resource orchestrator 1112 may implement any number and type of self-modifying actions based on any amount and type of data collected in any technically feasible manner. Self-modifying actions may relieve a data center operator of data center 1100 from making potentially poor configuration decisions and potentially avoid underutilized and / or poorly performing sections of a data center.

[0266] Data center 1100 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, one or more machine learning models may be trained by calculating weighting parameters according to a neural network architecture using software and / or computing resources described above with reference to data center 1100.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with reference to data center 1100 using weighting parameters calculated by one or more training techniques, such as, but not limited to, those described herein.

[0267] In at least one embodiment, data center 1100 may utilize CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inferencing using the resources described above. Furthermore, one or more of the software and / or hardware resources described above may be configured as a service to enable users to train or infer information, such as image capture, speech capture, or other artificial intelligence services. EXAMPLE NETWORK ENVIRONMENTS

[0268] Network environments suitable for implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of the one or more computing devices 1000 of Fig. 10 - for example, each device may include similar components, features, and / or functionality of the one or more computing devices 1000. If backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may also be included as part of a data center 1100, an example of which is described herein with reference to Fig. 11 is described in more detail.

[0269] The components of a network environment can communicate with each other over one or more networks, which can be wired, wireless, or both. The network can contain multiple networks or a network of networks. For example, the network can contain one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or a public switched telephone network (PSTN), and / or one or more private networks. If the network contains a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0270] Compatible network environments may include one or more peer-to-peer network environments—in which case, a server cannot be included in a network environment—and one or more client-server network environments—in which case, one or more servers can be included in a network environment. In peer-to-peer network environments, the functionality described herein with respect to one or more servers may be implemented on any number of client devices.

[0271] In at least one embodiment, a network environment may include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. A framework layer may include a framework for supporting software of a software layer and / or one or more applications of an application layer. The software or the one or more applications may each include web-based service software or applications. In embodiments, one or more of the client devices may utilize the web-based service software or applications (e.g.,by accessing the service software and / or applications through one or more application programming interfaces (APIs). The framework layer may be some type of free and open-source software web application framework, e.g., using, but not limited to, a distributed file system for processing large amounts of data (e.g., "Big Data").

[0272] A cloud-based network environment may provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Each of these various functions may be distributed from central or core servers (e.g., from one or more data centers that may be located across a state, region, country, globe, etc.) across multiple locations. When a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers may offload at least a portion of the functionality to the one or more edge servers. A cloud-based network environment may be private (e.g., restricted to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0273] The one or more client devices may include at least some of the components, features, and functions of the one or more described herein with respect to Fig.10. By way of example, and not limitation, a client device may be embodied as a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or global positioning device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, a hydrofoil, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or gaming system, an entertainment system, a vehicle computing system, an embedded system controller, a remote control, an appliance, a consumer electronics device, a workstation, an edge device,any combination of these described devices or any other suitable device.

[0274] The disclosure of this application also contains the following numbered clauses: Clause 1. Procedure, comprising: Receiving road data representing road segments and lane data representing a lane associated with a location of an ego machine; Determining that consecutive road segments correspond to a travel lane based on at least a geometric similarity between the travel lane and the consecutive road segments; and generating a representation of the lane assigned to the successive road segments based at least on determining that the successive road segments have been aligned with the lane; and Performing one or more operations corresponding to the ego machine based at least on the representation of the lane assigned to the successive road segments. Clause 2. The method of Clause 1, wherein determining that the successive road segments have been matched to the travel lane based at least on geometric similarity comprises determining that the travel lane is positioned within a distance threshold of one or more points along the successive road segments. Clause 3. The method of any preceding clause, wherein determining the successive road segments that coincide with the travel lane based at least on geometric similarity comprises determining that the travel lane is positioned within a directional threshold of one or more points along the successive road segments. Clause 4. The method of any preceding clause, wherein determining the successive road segments that coincide with the travel lane based at least on geometric similarity comprises determining that the travel lane is positioned within a distance threshold and a direction threshold from one or more points along the successive road segments. Clause 5. The method of any preceding clause, wherein determining the successive road segments that coincide with the travel lane based at least on geometric similarity comprises iteratively determining that one or more points along the successive road segments are positioned within a distance threshold of the travel lane based at least on extending one or more perpendicular line segments from the one or more points along the successive road segments to intersect the travel lane. Clause 6. A method according to clause 5, wherein the one or more points are positioned along the successive road sections at a predetermined distance between successive points of the one or more points. Clause 7. The method of any preceding clause, wherein determining the successive road segments that match the lane based at least on geometric similarity comprises determining that the lane is positioned within a distance threshold and a direction threshold for at least a predetermined number of points along the successive road segments. Clause 8. The method of any preceding clause, wherein determining the successive road segments that coincide with the travel lane based at least on the geometric similarity comprises determining that the travel lane is positioned within a distance threshold and a direction threshold for one or more points positioned along the successive road segments for at least a predetermined distance. Clause 9. The method of any preceding clause, further comprising generating a representation of an extent to which the successive road segments coincide with the travel lane based at least on the geometric similarity between the travel lane and the successive road segments. Clause 10. A method according to any preceding clause, wherein generating the representation of the extent to which the successive road segments coincide with the travel lane is based on at least a sum of one or more inverses of one or more distances between the travel lane and one or more sampled locations along the successive road segments. Clause 11. The method of any preceding clause, further comprising determining to generate the representation of the lane assigned to the successive road segments based at least on one representation of an extent to which the successive road segments coincide with the lane being greater than another representation of an extent to which other successive road segments coincide with the lane. Clause 12. A method according to any preceding clause, wherein the representation of the lane allocated to the successive road segments comprises a representation of a start and end location of a segment along the lane, a representation of an identity of at least some of the successive road segments, and a representation of a start and end location of at least one road segment. Clause 13. The method of any preceding clause, further comprising selecting the lane based at least on the lane having a position along its geometry within a maximum distance from the location of the ego machine. Clause 14. A method according to any preceding clause, wherein the method is carried out using at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing real-time streaming; a system for generating or presenting one or more augmented reality contents, Virtual Reality content or Mixed Reality content; a system for performing digital twin operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for generating synthetic data; or a system implemented at least in part using cloud computing resources. Clause 15. One or more processors comprising processing circuitry for: determining that one or more consecutive road segments detected by an ego machine match a lane detected by the ego machine based at least on a geometric similarity between the lane and the one or more consecutive road segments; and generating a representation of the lane assigned to the one or more consecutive road segments based at least on determining that the one or more consecutive road segments have been matched to the lane. Clause 16. The one or more processors of Clause 15, wherein determining the one or more consecutive road segments that match the travel lane based at least on geometric similarity comprises an iterative process of determining that the travel lane is positioned within a distance threshold and a direction threshold of a sequence of one or more points along the one or more consecutive road segments. Clause 17. The one or more processors of any of Clauses 15 or 16, wherein the one or more processors comprise at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing operations using conversational AI; a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system for generating synthetic data; a system for generating synthetic data using AI; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources. Clause 18. System comprising one or more processors for: Determining that a road segment of a road graph associated with an ego machine matches a lane of a lane graph associated with the ego machine based on at least one geometric similarity between the lane and the road segment; Generating a representation of an extent to which the road segment coincides with the lane based at least on the geometric similarity between the lane and the road segment; Generating a representation of the lane assigned to the road segment based at least on the representation of the extent to which the road segment coincides with the lane; and Performing one or more operations associated with controlling the ego machine based at least on the representation of the lane assigned to the road segment. Clause 19. A system according to Clause 18, wherein the representation of the extent to which the road segment coincides with the travel lane is generated based on at least a sum of one or more inverses of one or more distances between the travel lane and one or more sampled locations along the road segment. Clause 20. A system under either Clause 18 or 19, which system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing operations using conversational AI; a system for generating synthetic data; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources.

[0275] The disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., and refer to code that performs specific tasks or implements specific abstract data types. The disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc.The disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices that are interconnected via a network for communication.

[0276] As used herein, any reference to "and / or" in reference to two or more elements should be interpreted to mean only one element or a combination of elements. For example, "Element A, Element B, and / or Element C" may include only Element A, only Element B, only Element C, Element A and Element B, Element A and Element C, Element B and Element C, or Elements A, B, and C. Furthermore, "at least one of Element A or Element B" may include at least one of Element A, at least one of Element B, or at least one of Element A and at least one of Element B. Further, "at least one of Element A and Element B" may include at least one of Element A, at least one of Element B, or at least one of Element A and at least one of Element B.

[0277] The subject matter of the present disclosure is specifically described herein to satisfy legal requirements. However, the description itself is not intended to limit the scope of the present disclosure. Rather, the inventors have contemplated that the claimed subject matter may be embodied in other ways to include various steps or combinations of steps similar to those described herein, in conjunction with other present or future technologies. Although the terms "step" and / or "block" may be used herein to refer to various elements of the methods employed, the terms should not be interpreted to imply any particular ordering among or between the various steps disclosed herein, unless and except that the order of the individual steps is expressly described.

[0278] It is to be understood that aspects and embodiments described above are purely exemplary and that modifications of details may be made within the scope of the claims.

[0279] Each device, method, and feature disclosed in the description, and (where appropriate) the claims and drawings may be provided independently or in any suitable combination.

[0280] Reference signs appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 16 / 101,232

[0179] Cited non-patent literature

[0000] Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” of the Society of Automotive Engineers (SAE) (Standard No. J3016-201806, published on 15 June 2018, Standard No. J3016-201609, published on 30 September 2016

[0137]

Claims

[1] Method comprising: Receiving road data representing road segments and lane data representing a lane associated with a location of an ego machine; Determining that consecutive road segments correspond to a lane based on at least a geometric similarity between the lane and the consecutive road segments; and Generating a representation of the lane assigned to the successive road segments based at least on determining that the successive road segments have been aligned with the lane; and Performing one or more operations corresponding to the ego machine based at least on the representation of the lane assigned to the successive road segments. [2] The method of claim 1, wherein determining that the successive road segments have been matched to the travel lane based at least on the geometric similarity comprises determining that the travel lane is positioned within a distance threshold of one or more points along the successive road segments. [3] The method of any preceding claim, wherein determining the successive road segments that coincide with the lane based at least on the geometric similarity comprises determining that the lane is positioned within a directional threshold of one or more points along the successive road segments. [4] A method according to any one of the preceding claims, wherein determining the successive road segments that match the lane based at least on the geometric similarity comprises determining that the lane is positioned within a distance threshold and a direction threshold of one or more points along the successive road segments. [5] The method of any preceding claim, wherein determining the successive road segments that coincide with the travel lane based at least on the geometric similarity comprises iteratively determining that one or more points along the successive road segments are positioned within a distance threshold of the travel lane based at least on extending one or more perpendicular line segments from the one or more points along the successive road segments to intersect the travel lane. [6] The method of claim 5, wherein the one or more points are positioned along the successive road sections at a predetermined distance between successive points of the one or more points. [7] The method of any preceding claim, wherein determining the successive road segments that match the lane based at least on the geometric similarity comprises determining that the lane is positioned within a distance threshold and a direction threshold for at least a predetermined number of points along the successive road segments. [8] The method of any preceding claim, wherein determining the successive road segments that match the lane based at least on the geometric similarity comprises determining that the lane is positioned within a distance threshold and a direction threshold for one or more points positioned along the successive road segments for at least a predetermined distance. [9] The method of any preceding claim, further comprising generating a representation of an extent to which the successive road segments coincide with the lane based at least on the geometric similarity between the lane and the successive road segments. [10] A method according to any one of the preceding claims, wherein generating the representation of the extent to which the successive road segments coincide with the lane is based on at least a sum of one or more inverses of one or more distances between the lane and one or more sampled locations along the successive road segments. [11] The method of any preceding claim, further comprising determining to generate the representation of the lane assigned to the successive road segments based at least on one representation of an extent to which the successive road segments coincide with the lane being greater than another representation of an extent to which other successive road segments coincide with the lane. [12] A method according to any one of the preceding claims, wherein the representation of the lane allocated to the successive road sections comprises a representation of a start and end location of a section along the lane, a representation of an identity of at least a portion of the successive road sections, and a representation of a start and end location of at least one road section. [13] The method of any preceding claim, further comprising selecting the lane based at least on the lane having a position along its geometry within a maximum distance from the location of the ego machine. [14] A method according to any one of the preceding claims, wherein the method is carried out using at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system for performing digital twin operations; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for generating synthetic data; or a system implemented at least in part using cloud computing resources. [15] One or more processors comprising processing circuitry for: Determining that one or more consecutive road segments detected by an ego machine match a lane detected by the ego machine based on at least one geometric similarity between the lane and the one or more consecutive road segments; and Generating a representation of the lane assigned to the one or more consecutive road segments based at least on determining that the one or more consecutive road segments have been matched to the lane. [16] The one or more processors of claim 15, wherein determining the one or more consecutive road segments that match the travel lane based at least on the geometric similarity comprises an iterative process of determining that the travel lane is positioned within a distance threshold and a direction threshold of a sequence of one or more points along the one or more consecutive road segments. [17] The one or more processors of any one of claims 15 or 16, wherein the one or more processors comprise at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing operations using conversational AI; a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system for generating synthetic data; a system for generating synthetic data using AI; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources. [18] System comprising one or more processors for: Determining that a road segment of a road graph associated with an ego machine matches a lane of a lane graph associated with the ego machine based on at least one geometric similarity between the lane and the road segment; Generating a representation of an extent to which the road segment coincides with the lane based at least on the geometric similarity between the lane and the road segment; Generating a representation of the lane assigned to the road segment based at least on the representation of the extent to which the road segment coincides with the lane; and Performing one or more operations associated with controlling the ego machine based at least on the representation of the lane assigned to the road segment. [19] The system of claim 18, wherein the representation of the extent to which the road segment coincides with the travel lane is generated based on at least a sum of one or more inverses of one or more distances between the travel lane and one or more sampled locations along the road segment. [20] A system according to any one of claims 18 or 19, wherein the system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulations; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing operations using conversational AI; a system for generating synthetic data; a system that contains one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system implemented at least in part using cloud computing resources.

Images